Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier2): bound embedding chunks and preserve source evidence (#227)
Christopher Kevinee3ecab34786
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d5b76d88..25f8d015 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -134,106 +134,12 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Local-mode verifier uploads now copy directly to the absent destination,- avoiding an unnecessary rollback snapshot of an empty tests directory.- Inputs and rewritten scripts are prepared privately before publication,- so failed uploads preserve existing verifier files. Missing or invalid- sources fail rather than creating an empty verifier directory. Per-step- verifier uploads retain shared test files, with step files taking precedence.- After any command runs, the next upload replaces the tests directory, so- files or links the agent left there cannot change verification.-- Harbor's standalone verifier now runs on Python 3.9 task images after sharing- deterministic checks with the host evaluator. Packaged verifier checks execute- on a real Python 3.9 interpreter in CI to retain this compatibility.-- Plain removal of recognized regenerable caches no longer receives the Tier 3- destructive-command penalty. Dynamic commands, mixed targets and other- destructive operations retain their existing classification in both verifiers.--- Rubric evaluation preserves the original `SKILL.md` text without adding YAML- separators and includes reachable local Markdown guides under their source- paths. Required guidance is collected before optional previews, with cycle- detection, secure contained reads, verified file-alias deduplication, a- 256-document ceiling, and the existing- 3,000-byte per-file / 30,000-byte supplementary limits. Missing, unsafe, or- oversized required guidance stops judging without a score or provider request- ([#233](https://github.com/NVIDIA/SkillEvaluator/issues/233)).--- PII credential findings mask their values and source lines. GitHub CI stages- scanner output privately, redacts recognized credential values from retained- reports and logs, and withholds detailed output when redaction cannot finish.-- Local Claude Code trials retain Harbor's instruction environment variables so- prompts reach `claude --print`. The exception accepts only the legacy- `HARBOR_INSTRUCTION` name and Claude Code's UUID-named instruction variable;- protected paths, loader controls, and unrelated Harbor variables retain their- existing filtering ([#245](https://github.com/NVIDIA/SkillEvaluator/issues/245)).--- Tier 3 reports preserve unavailable evaluator lift as `null` when no baseline- ran, rather than reporting a measured zero. CLI evaluator tables display- missing or unavailable scores as `n/a` without interrupting JSON and HTML- report generation; measured zeroes retain their numeric formatting- ([#240](https://github.com/NVIDIA/SkillEvaluator/issues/240)).--- Explicit searches of `/` (`find`, `grep -r`, `rg`, `ag`, `du` and `tree` with a- `/` operand, and native search tools rooted at `/`) now count as wasted tool- calls whether or not they name the expected skill or script, including when- output is redirected or the search runs inside `sh -c`. Host and bundled- scoring agree, while scoped searches and slash-valued patterns retain their- classification.-- PII email scans recognize all RFC 2606 reserved domains, including their- subdomains, mixed-case forms, trailing dots, and prose ellipses. A local-part- boundary prevents quadratic rescanning of long excluded addresses. Ordinary- addresses on the same line and domains that merely resemble placeholders remain detectable- ([#244](https://github.com/NVIDIA/SkillEvaluator/issues/244)).--- Tier 3 security grading preserves ordinary nested `/tmp` cleanup, including- child operands with trailing slashes, unless the current or an earlier- execution action contains recognized symlink-creation syntax (`ln -s`,- `cp -s`, `os.symlink`, `Path.symlink_to`, literal Python subprocess argv- launching `ln`/`cp`, or `fs.symlink`/`fs.symlinkSync`). With that evidence,- both graders withhold nested-cleanup exemptions and report critical- destructive-command findings with a security score of 0.0; unlinking a- final leaf remains exempt. This conservative signal applies across the- trace, including after link moves or replacements, without inspecting the- filesystem or proving a link's destination or creation success. Unobserved- or pre-existing symlinks are not detected by this policy. Lexical path escapes- and unresolved or `xargs`-supplied deletion targets remain flagged. Searches,- script arguments, and unexecuted heredoc data do not supply link evidence;- Python and Node call syntax is inspected only in executable inline source- or a recognized program supplied directly through standard input. Comments- containing apostrophes do not hide actual link calls. Later and multiple- heredocs, `cat` pipelines into interpreters, and common shell startup options- retain link detection. Plain `;`, `&&`, `||`, and newline sequences apply- evidence in command order, so later link creation does not regrade earlier- cleanup in the same call. Loops, functions, traps, background jobs and other- compound syntax keep conservative whole-action evidence; unrelated link- creation can still affect later nested cleanup and remains a policy tradeoff.-- Tier 3 security grading no longer grants temporary-cleanup exemptions to- destructive `rm` commands launched through `xargs`, whose additional targets- may come from standard input or an argument file. Both graders recognize- common options and wrappers while preserving separate cleanup commands.- Input contents are not evaluated, so this conservative policy also flags- legitimate scoped cleanup such as- `find /tmp/cache -name '*.pyc' | xargs rm -f` as a critical destructive-command- finding with a security score of 0.0.-- Tier 3 security grading now normalizes deletion targets before exempting- temporary-directory cleanup. Paths such as `/tmp/../workspace` and commands- mixing temporary and non-temporary targets receive destructive-command- findings in both the host checker and bundled Harbor verifier. Destructive- `rm` fragments with no parsed target are also flagged, preserving detection- of deletions assembled from strings. Temporary cleanup inside backtick- command substitutions, or followed by shell comments and file-descriptor- redirections, no longer produces false destructive-command findings. Targets- after heredoc and here-string redirections are still checked; the input word- of a redirection is not treated as a deletion target.- Case-arm/subshell terminators and unrelated comments, ANSI-C literals, or- heredoc data no longer remove the cleanup exemption. Command substitutions- used as redirection words do not hide later deletion targets. Unquoted- `/tmp` deletion targets containing variables, command substitutions, or- brace expansions (for example standalone `rm -rf /tmp/build-$$`) no longer- get the cleanup exemption because their destinations remain unresolved.- Operandless `xargs rm` invocations remain detectable when unrelated syntax- or parser budgets prevent a complete parse; the fallback uses a bounded- command span to avoid excessive backtracking on long actions.+- Tier 2 caps every intra-skill embedding chunk at 3,000 characters, preserves+ meaningful short tails and actual Python docstring source ranges, and avoids+ counting pieces of one original chunk as duplicates of each other. Oversized+ Markdown sections are packed by paragraph, so short labels and fragments stay+ attached to their neighbours, and findings list a split source once.+- Tier 3 error recovery now scores unrecovered command failures as zero andincludes them in fault counts and failure verdicts/report reasons. Host andbundled verifier detection agree and use a result's execution status when itdiff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex 66c63d81..8e72f9ea 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -113,7 +113,7 @@ vocabulary.| Extension | Chunking strategy || --- | --- |-| `.md`, `.mdc` | Heading-based sections; sections over 3,000 characters are split at eligible paragraph boundaries when possible; chunks under 80 characters are dropped |+| `.md`, `.mdc` | Heading-based sections; eligible sections over 3,000 characters are split into bounded pieces, preferring paragraph and line boundaries || `.py` | Module docstrings plus class and function signatures with their docstrings; implementation bodies are not included || `.sh` | Function bodies and comment blocks of at least three consecutive comment lines || Other extensions | Skipped |@@ -123,6 +123,16 @@ empty, or non-mapping frontmatter remains part of the analyzed text. Everyretained chunk keeps its source file and line range, so findings point you tothe exact overlapping sections.+Each embedding input is capped at 3,000 characters, including long table rows,+Python docstrings, and shell functions. Oversized Markdown sections are packed+paragraph by paragraph up to the cap, and paragraphs under 80 characters stay+attached to a neighbouring paragraph rather than becoming chunks of their own.+Pieces split from one section, docstring, or function do not count as+duplicates of each other. An independent copy can still join a duplicate+cluster, and the finding then lists the split source once with its merged line+range. Python pieces retain the enclosing source range because cleaned+docstrings and function signatures are not contiguous text.+### Verdicts and what to do about them| Verdict | Meaning | Your action |@@ -283,8 +293,8 @@ you intend to skip Tier 2 entirely.| Inter-skill reporting threshold | `0.75` | Reports `SIMILAR` and stronger matches || Intra-skill clustering threshold | `0.80` | Selects candidate chunk pairs for clustering || Full-body chunking | 512 tokens, 64-token overlap | Chunked average pooling for `--full-body` embeddings |-| Minimum content chunk | 80 characters | Drops fragments too small to compare usefully |-| Large Markdown section split point | 3,000 characters | Splits oversized sections at eligible paragraph boundaries |+| Minimum content chunk | 80 characters | Drops sections and chunks too small to compare usefully; short paragraphs of an oversized section stay attached to a neighbour |+| Embedding input cap | 3,000 characters | Caps every Markdown, Python, and shell embedding input; oversized Markdown sections are packed at paragraph boundaries || High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM |Tier 2 treats skill content and catalog files as untrusted input. Before anydiff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.pyindex 2d68127e..d25f72bd 100644--- a/src/skillevaluator/constants.py+++ b/src/skillevaluator/constants.py@@ -249,6 +249,9 @@ CONTENT_DEDUP_SIMILARITY_THRESHOLD = 0.80CONTENT_DEDUP_MIN_CHUNK_CHARS = 80+# Bound each provider embedding input, including dense tables and long lines.+CONTENT_DEDUP_MAX_CHUNK_CHARS = 3000+# Upper bound (in characters) for treating a repeated chunk as a "trivial"# duplicate. Short, single-file blocks dominated by comment/config-style lines# (e.g. a recurring `# default pts-tolerance is 60 ms.` config snippet) arediff --git a/src/skillevaluator/deduplication/intra_skill/semantic_clustering.py b/src/skillevaluator/deduplication/intra_skill/semantic_clustering.pyindex 482e73a9..4287fde4 100644--- a/src/skillevaluator/deduplication/intra_skill/semantic_clustering.py+++ b/src/skillevaluator/deduplication/intra_skill/semantic_clustering.py@@ -9,7 +9,7 @@ similarity >= threshold to at least one other member in the group.from __future__ import annotations-from dataclasses import dataclass, field+from dataclasses import dataclass, field, replacefrom itertools import combinationsfrom typing import TYPE_CHECKING@@ -61,6 +61,28 @@ class UnionFind:return groups+def _merge_split_pieces(members: list[ContentChunk]) -> list[ContentChunk]:+ """Report pieces of one split source chunk once, with their merged range."""+ merged: list[ContentChunk] = []+ positions: dict[tuple[str, int, int], int] = {}+ for chunk in members:+ group = chunk.split_group+ if group is None:+ merged.append(chunk)+ elif group in positions:+ first = merged[positions[group]]+ merged[positions[group]] = replace(+ first,+ text=f"{first.text}\n\n{chunk.text}",+ start_line=min(first.start_line, chunk.start_line),+ end_line=max(first.end_line, chunk.end_line),+ )+ else:+ positions[group] = len(merged)+ merged.append(chunk)+ return merged++def build_clusters(chunks: list[ContentChunk],threshold: float = CONTENT_DEDUP_SIMILARITY_THRESHOLD,@@ -70,6 +92,10 @@ def build_clusters(if n < 2:return []+ def siblings(i: int, j: int) -> bool:+ group = chunks[i].split_group+ return group is not None and group == chunks[j].split_group+uf = UnionFind(n)pair_scores: dict[tuple[int, int], float] = {}@@ -79,7 +105,7 @@ def build_clusters(chunks[j].embedding,)pair_scores[(i, j)] = score- if score >= threshold:+ if score >= threshold and not siblings(i, j):uf.union(i, j)clusters: list[ContentCluster] = []@@ -87,8 +113,8 @@ def build_clusters(if len(indices) < 2:continue- members = [chunks[i] for i in indices]- scores = [pair_scores.get((min(i, j), max(i, j)), 0.0) for i, j in combinations(indices, 2)]+ members = _merge_split_pieces([chunks[i] for i in indices])+ scores = [pair_scores[(i, j)] for i, j in combinations(indices, 2) if not siblings(i, j)]clusters.append(ContentCluster(diff --git a/src/skillevaluator/deduplication/utils/chunker.py b/src/skillevaluator/deduplication/utils/chunker.pyindex f88340e7..4c3f98a4 100644--- a/src/skillevaluator/deduplication/utils/chunker.py+++ b/src/skillevaluator/deduplication/utils/chunker.py@@ -14,12 +14,12 @@ from __future__ import annotationsimport astimport loggingimport re-from dataclasses import dataclass, field+from dataclasses import dataclass, field, replacefrom typing import TYPE_CHECKINGfrom markdown_it import MarkdownIt-from skillevaluator.constants import CONTENT_DEDUP_MIN_CHUNK_CHARS+from skillevaluator.constants import CONTENT_DEDUP_MAX_CHUNK_CHARS, CONTENT_DEDUP_MIN_CHUNK_CHARSfrom skillevaluator.spdx import is_spdx_only_html_commentif TYPE_CHECKING:@@ -42,6 +42,8 @@ class ContentChunk:text: strsource_format: str # "markdown" | "python" | "shell"embedding: list[float] = field(default_factory=list)+ # Pieces of one source chunk must not create duplicate findings with each other.+ split_group: tuple[str, int, int] | None = None@propertydef char_count(self) -> int:@@ -55,12 +57,63 @@ def chunk_file("""Route to the appropriate chunker based on file extension."""ext = collected.extensionif ext in {".md", ".mdc"}:- return chunk_markdown(collected, min_chars)- if ext == ".py":- return chunk_python(collected, min_chars)- if ext == ".sh":- return chunk_shell(collected, min_chars)- return []+ chunks = chunk_markdown(collected, min_chars)+ elif ext == ".py":+ chunks = chunk_python(collected, min_chars)+ elif ext == ".sh":+ chunks = chunk_shell(collected, min_chars)+ else:+ return []+ return [piece for chunk in chunks for piece in _bounded_pieces(chunk)]+++def _bounded_pieces(chunk: ContentChunk) -> list[ContentChunk]:+ """Cap provider inputs, keeping meaningful tails and source containment.++ The minimum-length filter applies to the original chunk. Its final short+ piece still contains evidence and must not disappear during splitting.+ Python text joins signatures and cleaned docstrings rather than contiguous+ source lines, so every piece retains the enclosing AST node's range.+ """+ limit = CONTENT_DEDUP_MAX_CHUNK_CHARS+ if len(chunk.text) <= limit:+ return [chunk]+ pieces: list[ContentChunk] = []+ group = chunk.split_group or (chunk.source_file, chunk.start_line, chunk.end_line)+ buffer = ""+ buffer_start = 0++ def add(text: str, first: int, last: int) -> None:+ if not text.strip():+ return+ start = min(chunk.start_line + first, chunk.end_line)+ end = min(chunk.start_line + last, chunk.end_line)+ if chunk.source_format == "python":+ start, end = chunk.start_line, chunk.end_line+ pieces.append(+ replace(chunk, text=text.strip(), start_line=start, end_line=end, embedding=[], split_group=group)+ )++ def flush(last: int) -> None:+ nonlocal buffer+ if buffer:+ add(buffer, buffer_start, last)+ buffer = ""++ lines = chunk.text.splitlines(keepends=True)+ for offset, line in enumerate(lines):+ if len(line) > limit:+ flush(max(0, offset - 1))+ for index in range(0, len(line), limit):+ add(line[index : index + limit], offset, offset)+ else:+ if len(buffer) + len(line) > limit:+ flush(max(0, offset - 1))+ if not buffer:+ buffer_start = offset+ buffer += line+ flush(len(lines) - 1)+ return piecesdef chunk_markdown(@@ -85,15 +138,15 @@ def chunk_markdown(match = None if i in fenced_lines else HEADING_PATTERN.match(line)if match:if current_lines:- text = "".join(current_lines).strip()- if len(text) >= min_chars and not (- current_heading == "(preamble)" and is_spdx_only_html_comment(text)- ):+ raw_text = "".join(current_lines)+ text = raw_text.strip()+ leading_lines = raw_text[: len(raw_text) - len(raw_text.lstrip())].count("\n")+ if len(text) >= min_chars and not (current_heading == "(preamble)" and is_spdx_only_html_comment(text)):sections.append(ContentChunk(source_file=collected.rel_path,heading=current_heading,- start_line=current_start,+ start_line=current_start + leading_lines,end_line=i - 1,text=text,source_format="markdown",@@ -106,15 +159,15 @@ def chunk_markdown(current_lines.append(line)if current_lines:- text = "".join(current_lines).strip()- if len(text) >= min_chars and not (- current_heading == "(preamble)" and is_spdx_only_html_comment(text)- ):+ raw_text = "".join(current_lines)+ text = raw_text.strip()+ leading_lines = raw_text[: len(raw_text) - len(raw_text.lstrip())].count("\n")+ if len(text) >= min_chars and not (current_heading == "(preamble)" and is_spdx_only_html_comment(text)):sections.append(ContentChunk(source_file=collected.rel_path,heading=current_heading,- start_line=current_start,+ start_line=current_start + leading_lines,end_line=collected.line_offset + len(lines),text=text,source_format="markdown",@@ -123,14 +176,36 @@ def chunk_markdown(result: list[ContentChunk] = []for section in sections:- if len(section.text) > 3000:- result.extend(split_into_paragraphs(section, min_chars))+ if len(section.text) > CONTENT_DEDUP_MAX_CHUNK_CHARS:+ result.extend(_pack_paragraphs(section, min_chars))else:result.append(section)return result+def _pack_paragraphs(section: ContentChunk, min_chars: int) -> list[ContentChunk]:+ """Pack consecutive paragraphs of an oversized section up to the embedding cap.++ Paragraphs shorter than ``min_chars`` stay attached to a neighbour instead of+ becoming chunks of their own; ``_bounded_pieces`` caps any overflow. Every+ piece shares the section's split group, so pieces never cluster together.+ """+ group = (section.source_file, section.start_line, section.end_line)+ packed: list[ContentChunk] = []+ for paragraph in split_into_paragraphs(section, min_chars=1):+ if packed and (+ len(packed[-1].text) + 2 + len(paragraph.text) <= CONTENT_DEDUP_MAX_CHUNK_CHARS+ or len(paragraph.text) < min_chars+ or len(packed[-1].text) < min_chars+ ):+ last = packed[-1]+ packed[-1] = replace(last, text=f"{last.text}\n\n{paragraph.text}", end_line=paragraph.end_line)+ else:+ packed.append(replace(paragraph, split_group=group))+ return packed++def split_into_paragraphs(section: ContentChunk,min_chars: int,@@ -182,12 +257,13 @@ def chunk_python(module_doc = ast.get_docstring(tree)if module_doc and len(module_doc) >= min_chars:+ doc_node = tree.body[0]chunks.append(ContentChunk(source_file=collected.rel_path,heading="(module docstring)",- start_line=1,- end_line=module_doc.count("\n") + 2,+ start_line=doc_node.lineno,+ end_line=doc_node.end_lineno or doc_node.lineno,text=module_doc,source_format="python",)diff --git a/tests/deduplication/utils/test_chunk_bounds.py b/tests/deduplication/utils/test_chunk_bounds.pynew file mode 100644index 00000000..a99ca4ed--- /dev/null+++ b/tests/deduplication/utils/test_chunk_bounds.py@@ -0,0 +1,104 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+"""Bound embedding inputs without losing source identity or inventing duplicates."""++from pathlib import Path++import pytest++from skillevaluator.deduplication.intra_skill.semantic_clustering import build_clusters+from skillevaluator.deduplication.utils.chunker import ContentChunk, chunk_file+from skillevaluator.deduplication.utils.skill_collector import CollectedFile+++def collected(text, extension=".md"):+ return CollectedFile(Path("example" + extension), "example" + extension, extension, text, len(text.splitlines()))++[email protected]("extension", [".md", ".mdc", ".py", ".sh"])+def test_every_embedding_input_is_bounded(extension):+ body = "measurement " * 5000+ if extension == ".py":+ text = '"""' + body + '"""\n'+ elif extension == ".sh":+ text = 'run() {\n echo "' + body + '"\n}\n'+ else:+ text = "# Reference\n| Key | Value |\n" + "| entry | " + body + "|\n"+ chunks = chunk_file(collected(text, extension))+ assert len(chunks) > 1+ assert max(len(c.text) for c in chunks) <= 3000+ assert all(c.start_line <= c.end_line for c in chunks)+++def test_split_preserves_a_short_meaningful_tail():+ chunks = chunk_file(collected("a" * 3000 + "TAIL"))+ assert "".join(c.text for c in chunks) == "a" * 3000 + "TAIL"+ assert max(len(c.text) for c in chunks) <= 3000+++def test_python_module_docstring_uses_actual_ast_lines():+ text = "\n" * 40 + '"""\n' + "documentation " * 20 + '\n"""\n'+ chunks = chunk_file(collected(text, ".py"))+ assert [(c.start_line, c.end_line) for c in chunks] == [(41, 43)]+++def test_split_python_signature_and_docstring_keep_the_containing_range():+ text = 'def run(\n argument,\n):\n """\n' + " documentation " * 500 + '\n """\n pass\n'+ chunks = chunk_file(collected(text, ".py"))+ assert len(chunks) > 1+ assert all((c.start_line, c.end_line) == (1, 7) for c in chunks)+++def test_sibling_pieces_do_not_create_duplicate_clusters():+ pieces = chunk_file(collected("reference " * 1500))+ for piece in pieces:+ piece.embedding = [1.0, 0.0]+ assert len(pieces) > 1+ assert build_clusters(pieces) == []+++def test_independent_copy_can_still_link_split_pieces():+ pieces = chunk_file(collected("reference " * 1500))+ for piece in pieces:+ piece.embedding = [1.0, 0.0]+ copy = ContentChunk("copy.md", "reference", 1, 1, pieces[0].text, "markdown", embedding=[0.9, 0.1])+ clusters = build_clusters([*pieces, copy])+ assert len(clusters) == 1+ assert clusters[0].cross_file+ assert clusters[0].max_similarity < 1.0+ assert len(clusters[0].members) == 2+ merged = next(member for member in clusters[0].members if member.split_group is not None)+ assert merged.text.count("reference") == sum(piece.text.count("reference") for piece in pieces)+++def test_fenced_headings_are_not_promoted_to_sections_during_split():+ text = "# Reference\n```text\n# example heading\n" + "x" * 7000 + "\n```\n"+ chunks = chunk_file(collected(text))+ assert len(chunks) > 1+ assert all(c.heading == "# Reference" for c in chunks)+++def test_markdown_preamble_ranges_follow_leading_blank_lines():+ chunks = chunk_file(collected("\n" * 40 + "instruction " * 700))+ assert len(chunks) > 1+ assert all((c.start_line, c.end_line) == (41, 41) for c in chunks)+++def test_oversized_section_preserves_short_final_paragraph():+ chunks = chunk_file(collected("# Guide\n" + "instruction " * 300 + "\n\nIMPORTANT_TAIL: Keep a backup."))+ assert any("IMPORTANT_TAIL: Keep a backup." in c.text for c in chunks)+++def test_oversized_section_keeps_short_labels_with_their_examples():+ example = (+ "Example:\n\n```bash\nskill run --input data.csv\n\nskill run --verbose\n```\n\n"+ "This example shows how the command reads its input and reports progress.\n\n"+ )+ chunks = chunk_file(collected("# Usage\n" + example * 25))+ assert len(chunks) > 1+ assert min(len(c.text) for c in chunks) >= 80+ assert max(len(c.text) for c in chunks) <= 3000+ assert len({c.split_group for c in chunks}) == 1+ for chunk in chunks:+ chunk.embedding = [1.0, 0.0]+ assert build_clusters(chunks) == []