Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
ci: keep the known failures running under a strict xfail (#1870)
Kevineba0abce3ff5
diff --git a/.github/full-suite-known-failures.txt b/.github/full-suite-known-failures.txtindex 30caa14d..84c680ce 100644--- a/.github/full-suite-known-failures.txt+++ b/.github/full-suite-known-failures.txt@@ -1,10 +1,24 @@# Tests that already fail on an unmodified tree.#-# The `Run the full test suite` step in .github/workflows/ci.yml deselects-# every node id in this file, so that step gates on new breakage instead of on-# these pre-existing defects. Nothing here is caused by wiring the suite into-# CI, and none of it is fixed here on purpose: the step reports the gap, it-# does not paper over it.+# `tests/conftest.py` reads this file and marks every id below `xfail`, so the+# tests keep running instead of being deselected. The marker is strict in CI and+# non-strict outside it, and never strict for a `# non-strict` entry: an entry+# that starts passing reports XPASS, which fails the run in CI and is only a note+# locally. In CI that is what forces its line out of the file, so the list cannot+# quietly rot. Nothing here is caused by wiring the suite into CI, and none of it+# is fixed here on purpose: the list reports the gap, it does not paper over it.+#+# Two markers change how an entry is treated, and both the step and the hook+# read them so the two agree:+#+# # windows-only the id fails only in the Windows measurement. The step+# deselects it on Windows alone, and the hook leaves it out+# on Linux, so there it runs and gates.+# # non-strict the hook marks the id xfail(strict=False), in CI as well as+# locally: it keeps running, but a pass is reported as XPASS+# without failing the run. For an entry that genuinely varies+# with run order or timing (a thread race, for instance);+# every other entry below is strict in CI.## Provenance. Generated on a clean checkout of#@@ -24,46 +38,19 @@# That measurement is not a superset of a Linux run: the entries appended below# from the suite's first Linux run are failures it could not see, so this list# is the union of the two runs rather than an upper bound from either. The-# entries marked `# windows-only` are expected to pass on Linux and are-# deselected on Windows only. The rest are expected to reproduce.+# entries marked `# windows-only` are expected to pass on Linux. The rest are+# expected to reproduce.#-# Format: one pytest node id per line. The step ignores a line only when it is-# empty or its first character is `#`, so keep comments flush left — an indented-# comment would be read as a node id and deselected as the empty string. A-# trailing `# windows-only` marker means the step deselects that id when it runs-# on Windows, and runs it everywhere else.+# Format: one pytest node id per line. A line is an id only when it is+# non-empty and its first character is not `#`, so keep comments flush left —+# an indented comment would be read as a node id. A trailing `# windows-only` or+# `# non-strict` marker is stripped before the id is used. The hook matches+# the id the way pytest writes it, relative to the rootdir (the repository root+# unless --rootdir or -c overrides it).## Delete a line to re-enable a test. Fixing the defect behind it is what should# drive that, not convenience.-# --- The test double patches a call the ingest path does not make --------------# Each of these stubs `requests` at a layer the ingestors do not use: they fetch-# through `request_with_ssrf_guard()`, which issues `session.request(...)` —-# neither `requests.get()` nor `Session.get()`. The stub therefore intercepts-# nothing, the request is really sent, and the guard-# pins the connection to the address it resolved. `tests/ingest/conftest.py`-# installs an autouse fixture that stubs `ssrf.socket.getaddrinfo` to map *every*-# hostname to the single literal 93.184.216.34, so all four calls aim at that-# one address regardless of the URL the test passes. Windows refuses to connect-# to it (`[WinError 10049] 在其上下文中,该请求的地址无效`); on the Linux runner-# the connection times out instead. They fail on both platforms, so they are-# deselected unconditionally rather than on Windows only.-# What each one stubs, and why it misses:-# test_ingest_feed — `requests.get`; the feed fetch goes through-# `request_with_ssrf_guard()`, so the stub misses and GET /feed.xml is sent.-tests/ingest/test_ingestors.py::TestFeedIngestor::test_ingest_feed-# test_ingest_url_happy — `requests.Session.get`; the page fetch calls-# `request_with_ssrf_guard(..., session=...)`, which uses `session.request`,-# so the stub misses and GET / is sent.-tests/ingest/test_web_ingestor.py::test_ingest_url_happy-# test_sitemap_fallback_parsing — `requests.get`; `SitemapCrawler.parse_sitemap`-# goes through `request_with_ssrf_guard()`, so the stub misses and GET / is sent.-tests/ingest/test_web_ingestor.py::test_sitemap_fallback_parsing-# test_sitemap_index_recursion — `requests.get`; `SitemapCrawler.crawl_sitemap_index`-# goes through `request_with_ssrf_guard()`, so the stub misses and GET /index.xml-# is sent.-tests/ingest/test_web_ingestor.py::test_sitemap_index_recursion-# --- Windows-only: symlink creation needs a privilege -------------------------# `OSError: [WinError 1314] 客户端没有所需的特权` — creating the symlink the# test needs requires Developer Mode or an elevated shell on Windows. On Linux@@ -78,26 +65,30 @@ tests/ingest/test_ingestors.py::TestUnifiedIngest::test_ingest_web_dispatchtests/ingest/test_ingestors.py::TestWebIngestor::test_ingest_url# --- Local embedding backend unavailable offline -------------------------------# `128 != 384` and `'fallback' != 'sentence_transformers'`: the real model could-# not be loaded in this environment, so the fallback backend answers instead.-tests/test_embedding_providers.py::TestEmbeddingProviders::test_embedding_generator_config-tests/test_embedding_providers.py::TestEmbeddingProviders::test_sentence_transformers_default-tests/test_model_selection.py::TestModelSelection::test_dynamic_switching-tests/test_model_selection.py::TestModelSelection::test_generator_switching+# The generator-config case is the exception: on the Linux runner the model does+# load and the test passes, so it is marked `# windows-only` rather than left to+# a strict xfail that would fail on that pass. The rest of this group is gone:+# #1931 restored the semantic_extract config isolation and they now pass.+tests/test_embedding_providers.py::TestEmbeddingProviders::test_embedding_generator_config # windows-only-# --- Order-dependent: these pass in isolation, fail in a full run --------------# The progress tracker is process-wide state; whichever test enables it first-# decides what the next one observes.+# --- Progress tracking is forced off for the whole process --------------------+# `semantica_mcp/mcp/__init__.py` sets `SEMANTICA_DISABLE_PROGRESS=1` at import+# and `tests/test_mcp_package_export_graphml_parquet.py` sets it at module scope+# as well, so the variable is already set while pytest collects the suite.+# `ProgressTracker.enabled` then latches disabled for the process+# (`progress_tracker.py:1140`) and the re-enable in `arrow_exporter.py:205` is a+# no-op. Both pass alone and fail in every full-suite run — collection of the+# MCP modules decides it, not the order — so they stay strict: fixing the+# isolation makes them pass and the XPASS forces these lines out.tests/test_arrow_exporter.py::TestArrowExporter::test_progress_trackingtests/vector_store/test_backward_compatibility.py::TestVectorStoreBackwardCompatibility::test_progress_tracking_unchanged# --- Assertion drift in the test or its expectation ---------------------------tests/export/test_distance_exporter.py::test_hop_distance_no_warning_when_kg_unavailable-tests/ingest/test_repo_ingestor_security.py::TestIsReservedNAT64Regression::test_nat64_local_prefix_not_blocked+tests/ingest/test_repo_ingestor_security.py::TestIsReservedNAT64Regression::test_nat64_local_prefix_not_blocked # windows-onlytests/integrations/agno/test_context_store.py::TestAgnoContextStoreUpsertNERAttributes::test_upsert_indexes_entity_label_as_typetests/integrations/agno/test_context_store.py::TestAgnoContextStoreUpsertNERAttributes::test_upsert_indexes_entity_text_not_repr-tests/semantic_extract/test_cache_backends.py::test_zero_ttl_expires_immediately_on_both_backends-tests/vector_store/test_hybrid_similarity.py::TestHybridSimilarityCalculator::test_pearson_similarity_calculation+tests/semantic_extract/test_cache_backends.py::test_zero_ttl_expires_immediately_on_both_backends # windows-onlytests/vector_store/test_vector_store.py::test_backend_mirroring_and_save# --- Integration tests outside the two dedicated integration directories ------@@ -109,10 +100,13 @@ tests/vector_store/test_vector_store.py::test_backend_mirroring_and_save# `sqlalchemy` extra this job does not install, a fallback backend answering# because the real embedding model is offline, and two expectation drifts in# the deepdive suite.+#+# `test_06_hybrid_search` used to sit here: it failed because `HybridSearch`+# dropped the `filter=` argument. #1827 fixed that, the test now passes on the+# runner, and its line is gone. `test_03_model_switching` was here too; #1931+# turned it green.tests/ingest/test_notebook_02.py::TestNotebook02DataIngestion::test_03_web_ingestiontests/ingest/test_notebook_02.py::TestNotebook02DataIngestion::test_08_database_ingestion-tests/test_all_features.py::TestSemanticaFeatures::test_03_model_switching-tests/test_all_features.py::TestSemanticaFeatures::test_06_hybrid_searchtests/test_semantic_extract_deepdive.py::TestSemanticExtractDeepDive::test_ner_extractor_patterntests/test_semantic_extract_deepdive_part2.py::TestSemanticExtractDeepDivePart2::test_coreference_resolver_pronounstests/test_semantic_extract_deepdive_part2.py::TestSemanticExtractDeepDivePart2::test_event_detector_basic@@ -144,6 +138,7 @@ tests/vector_store/test_faiss_index.py::test_builder_trains_pq_index# `expected call not found`: the call the Weaviate test records no longer# matches what the store issues. `assert 2 == 5`: the Jena empty-graph test# counts operations that all complete only once every concurrent writer has-# landed.+# landed. That one is a thread race and can pass when the writers happen to+# serialize, so it is `# non-strict`.tests/vector_store/test_backend_metadata_filtering.py::TestBackendMetadataFiltering::test_weaviate_store_filter_by_metadata_native_filter-tests/triplet_store/test_jena_store_empty_graph.py::TestJenaStoreEmptyGraph::test_empty_graph_with_concurrent_operations+tests/triplet_store/test_jena_store_empty_graph.py::TestJenaStoreEmptyGraph::test_empty_graph_with_concurrent_operations # non-strictdiff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 941b6c81..187eb61c 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -159,38 +159,58 @@ jobs:# two consistent.## .github/full-suite-known-failures.txt lists the tests that already fail- # on an unmodified checkout. They are deselected here, so this step gates- # on new breakage instead of on those pre-existing defects. The file is- # the record of the gap: fixing a defect means deleting its line. It is- # LF-only — the loop below feeds each line to --deselect verbatim, so a- # stray CR would make every node id miss.+ # on an unmodified checkout. tests/conftest.py marks those ids `xfail`, so+ # they keep running and an entry that starts passing reports XPASS and+ # fails the job — that is what forces its line out of the file. (The+ # marker is strict here because `CI` is set; outside CI the hook makes it+ # non-strict, since the list is measured against this pinned tree and not+ # against a developer's.) The loop below deselects only the entries the+ # hook must not touch: the `# windows-only` ones, and only on Windows. The+ # file is the record of the gap: fixing a defect means deleting its line.+ # It is kept LF-only (.gitattributes); the loop below trims whitespace+ # around each id, including a stray CR.- name: Run the full test suite+ timeout-minutes: 20run: |# A node id that no longer collects is a no-op for --deselect rather- # than an error, so a stale line cannot break the job; it just stops- # deselecting, and the test it named runs again.+ # than an error, so a stale line cannot break the job. It is inert for+ # the hook too, which only marks ids pytest actually collects: it+ # neither runs nor reports XPASS, and the file's own review is what+ # catches it.#- # A line may carry a trailing `# windows-only` marker. That means the- # id fails only on Windows and is deselected there alone — a symlink- # that needs a privilege, for instance. On Linux it runs, which is the- # only place the symlink rejection in+ # A line is deselected here only when it carries the `# windows-only`+ # marker and this is a Windows runner. Every other line reaches the+ # hook, which marks it xfail strictly here in CI.+ #+ # `# windows-only` is deselected on Windows alone — a symlink that+ # needs a privilege, for instance. On Linux the line is left out, so+ # the test runs, which is the only place the symlink rejection in# tests/semantic_extract/test_cache_backends.py gets exercised at all;# deselecting it unconditionally would drop a security check on the- # platform where it works.+ # platform where it works. This job runs on ubuntu-latest, so the+ # Windows branch never fires in CI today; it is kept for a Windows+ # runner, where that id does fail, rather than dropped as dead weight.deselect=()+ known=0+ deselected=0while IFS= read -r line; docase "$line" in''|'#'*) continue ;;- *windows-only*)+ esac+ known=$((known + 1))+ case "$line" in+ *'# windows-only'*)if [ "${RUNNER_OS:-}" != "Windows" ]; then continue; fi;;+ *) continue ;;esacnode="${line%%#*}"node="${node#"${node%%[![:space:]]*}"}"node="${node%"${node##*[![:space:]]}"}"deselect+=(--deselect "$node")+ deselected=$((deselected + 1))done < .github/full-suite-known-failures.txt- echo "deselecting known failures: ${#deselect[@]}"+ echo "known failures on the list: ${known}; deselected in this step: ${deselected}"# The google_adk and open_ontologies directories have their own steps# at the end of this job, each with its own dependency installation.diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.ymlindex 69f9d17b..02c54916 100644--- a/.github/workflows/codeql.yml+++ b/.github/workflows/codeql.yml@@ -34,7 +34,7 @@ jobs:# meaningful state carried over from a failed attempt.- name: Initialize CodeQL (attempt 1)id: codeql-init-1- uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/init@24c54180a607b1449ed407dd24f251e4e9147c8d # v4continue-on-error: truewith:languages: python@@ -44,7 +44,7 @@ jobs:- name: Initialize CodeQL (attempt 2)id: codeql-init-2if: steps.codeql-init-1.outcome == 'failure'- uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/init@24c54180a607b1449ed407dd24f251e4e9147c8d # v4continue-on-error: truewith:languages: python@@ -54,17 +54,17 @@ jobs:- name: Initialize CodeQL (attempt 3)id: codeql-init-3if: steps.codeql-init-2.outcome == 'failure'- uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/init@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:languages: pythonqueries: security-and-qualityconfig-file: .github/codeql/codeql-config.yml- name: Autobuild- uses: github/codeql-action/autobuild@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/autobuild@24c54180a607b1449ed407dd24f251e4e9147c8d # v4- name: Perform CodeQL Analysis- uses: github/codeql-action/analyze@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/analyze@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:category: "/language:python"upload: false@@ -74,7 +74,7 @@ jobs:# Uploads results only when Default Setup is not active.# If Default Setup is still enabled, this step skips gracefully# instead of failing the workflow with HTTP 409.- uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:sarif_file: ${{ steps.codeql.outputs.sarif-output }}category: "/language:python"diff --git a/.github/workflows/container-scan.yml b/.github/workflows/container-scan.ymlindex b74fafb6..3e9be255 100644--- a/.github/workflows/container-scan.yml+++ b/.github/workflows/container-scan.yml@@ -61,7 +61,7 @@ jobs:- name: Upload Trivy SARIFif: always()- uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:sarif_file: trivy-results.sarifcategory: trivy-containerdiff --git a/.github/workflows/defender-for-devops.yml b/.github/workflows/defender-for-devops.ymlindex 1a9a556f..8e5da485 100644--- a/.github/workflows/defender-for-devops.yml+++ b/.github/workflows/defender-for-devops.yml@@ -59,7 +59,7 @@ jobs:# avoiding the guardian.cmd/checkov exit-code bug in the MSDO wrapper.tools: eslint,templateanalyzer,terrascan- name: Upload results to Security tab- uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:sarif_file: ${{ steps.msdo.outputs.sarifFile }}@@ -100,7 +100,7 @@ jobs:run: python .github/scripts/filter_checkov_skipped.py reports/results_json.json reports/results_sarif.sarif reports/checkov.sarif- name: Upload Checkov results to Security tab- uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4if: always()with:sarif_file: reports/checkov.sarifdiff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.ymlindex 7530ab43..f19789c1 100644--- a/.github/workflows/scorecard.yml+++ b/.github/workflows/scorecard.yml@@ -40,6 +40,6 @@ jobs:retention-days: 5- name: Upload to code-scanning- uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4+ uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4with:sarif_file: results.sarifdiff --git a/tests/conftest.py b/tests/conftest.pyindex 2d432015..c16d45da 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -1,8 +1,41 @@-"""Shared pytest configuration for the whole test suite."""+"""+Suite-wide pytest configuration.++``.github/full-suite-known-failures.txt`` records the tests that already fail on+an unmodified checkout. Rather than deselecting them, the hook below marks the+listed ids ``xfail``: they keep running, and one that starts passing reports+XPASS — which, under a strict marker, fails the run and so forces its line out+of the file.++The list only holds against the pinned ``requirements-ci.txt`` tree it was+measured on. A developer machine has a different environment, so entries here+can pass there, and a strict marker would turn each of those passes into a+failure of an ordinary local run. The marker is therefore strict only when+``CI`` is set to a true value (``1``/``true``/``yes``; GitHub Actions sets+``CI=true`` for every job) and non-strict everywhere else: locally a listed entry still runs and is still xfailed, but a+pass is reported as XPASS without failing the run.++The ``Run the full test suite`` step in ``.github/workflows/ci.yml`` deselects+the entries it must — the ones marked ``# windows-only``, on Windows alone — so+those never reach this hook. The hook leaves a ``# windows-only`` entry out on+Linux, where it runs and gates. (For the symlink-rejection test that is the only+platform where the rejection is exercised at all.) The ``# non-strict`` marker keeps an+entry non-strict even in CI, for an entry that genuinely varies with run order+or timing; every other listed entry is strict in CI.++The autouse fixture below isolates the process-wide ``semantic_extract`` config+between tests: a double left there by one test otherwise makes every later+provider test see an API key that is not set, which is why the 24 ``test_llm_*``+ids used to fail in a full-suite run (#1913).+"""+from __future__ import annotationsimport copyimport importlib+import os+import reimport sys+from pathlib import Pathimport pytest@@ -26,3 +59,50 @@ def _isolate_semantic_extract_config():configs.clear()configs.update(saved)sys.modules[_CONFIG_MODULE_NAME] = _config_module+++_KNOWN_FAILURES = (+ Path(__file__).resolve().parent.parent+ / ".github"+ / "full-suite-known-failures.txt"+)++_WINDOWS_ONLY = "# windows-only"+_NON_STRICT = "# non-strict"+_HEADING = re.compile(r"^#\s*-{3,}\s+(?P<title>\S.*?)\s*-*\s*$")+_DEFAULT_REASON = "listed in .github/full-suite-known-failures.txt"+++def _listed_ids() -> dict[str, tuple[str, bool]]:+ """Return the ids to xfail, each mapped to its heading and strictness."""+ if not _KNOWN_FAILURES.is_file():+ return {}+ listed: dict[str, tuple[str, bool]] = {}+ reason = _DEFAULT_REASON+ for raw in _KNOWN_FAILURES.read_text(encoding="utf-8").splitlines():+ line = raw.strip()+ if not line:+ continue+ if line.startswith("#"):+ heading = _HEADING.match(line)+ if heading:+ reason = heading.group("title")+ continue+ if _WINDOWS_ONLY in line and sys.platform != "win32":+ continue+ node = line.split("#", 1)[0].strip()+ listed[node] = (reason, _NON_STRICT not in line)+ return listed+++def pytest_collection_modifyitems(config, items):+ listed = _listed_ids()+ if not listed:+ return+ in_ci = os.environ.get("CI", "").strip().lower() in {"1", "true", "yes"}+ for item in items:+ entry = listed.get(item.nodeid)+ if entry is None:+ continue+ reason, strict_ok = entry+ item.add_marker(pytest.mark.xfail(reason=reason, strict=in_ci and strict_ok))diff --git a/tests/ingest/test_ingestors.py b/tests/ingest/test_ingestors.pyindex da1392ee..44da923d 100644--- a/tests/ingest/test_ingestors.py+++ b/tests/ingest/test_ingestors.py@@ -78,8 +78,8 @@ class TestWebIngestor:assert "Test content" in result.textclass TestFeedIngestor:- @patch("requests.get")- def test_ingest_feed(self, mock_get):+ @patch("semantica.ingest.feed_ingestor.request_with_ssrf_guard")+ def test_ingest_feed(self, mock_request):ingestor = FeedIngestor()rss_content = """@@ -101,7 +101,7 @@ class TestFeedIngestor:mock_response.status_code = 200mock_response.text = rss_contentmock_response.content = rss_content.encode('utf-8')- mock_get.return_value = mock_response+ mock_request.return_value = mock_responseresult = ingestor.ingest_feed("http://example.com/feed.xml")diff --git a/tests/ingest/test_web_ingestor.py b/tests/ingest/test_web_ingestor.pyindex 4ce6d908..f63b0af9 100644--- a/tests/ingest/test_web_ingestor.py+++ b/tests/ingest/test_web_ingestor.py@@ -43,7 +43,9 @@ def test_sitemap_index_recursion() -> None:</urlset>"""- with patch("requests.get") as mock_get:+ with patch(+ "semantica.ingest.web_ingestor.request_with_ssrf_guard"+ ) as mock_get:mock_get.side_effect = [MagicMock(status_code=200, content=index_xml.encode()),MagicMock(status_code=200, content=child_xml.encode()),@@ -60,7 +62,7 @@ def test_sitemap_fallback_parsing() -> None:xml = "<urlset><url><loc>http://a.com</loc></url></urlset>"with patch(- "requests.get",+ "semantica.ingest.web_ingestor.request_with_ssrf_guard",return_value=MagicMock(status_code=200,content=xml.encode(),@@ -113,7 +115,7 @@ def test_extract_metadata_empty() -> None:# --- WebIngestor Tests ----@patch("requests.Session.get")+@patch("semantica.ingest.web_ingestor.request_with_ssrf_guard")def test_ingest_url_happy(mock_get: MagicMock, sample_html: str) -> None:"""Test successful URL ingestion."""