Skip to content

You're reading this. The skill predicted it. A workflow that turns every post into a calibrated experiment—score, blind-predict, retro, evolve. The future doesn't reward effort, it rewards those who see the pattern first. 1M followers in a month — not luck, system.

Python

fix: replace cheat-shoot line-level diff with char-level Levenshtein on normalized text (#14)

Jooonnn73536a4f355c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7152828..efb2d32 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,38 @@ All notable changes to cheat-on-content will be documented here.
## [Unreleased]
+### Fixed — cheat-shoot DIFF_METRIC 在口语化场景的 v2 误触发(**BREAKING for v2-trigger-logic**)
+
+**问题**:cheat-shoot Phase 3b 用 line-level unified diff 算 `diff_pct = (added + removed) * 100 / orig_lines`。但**创作者真实场景**——draft 是 markdown 长句(一行 ~50 字),拍摄稿是 whisper 转录的口语化短断句(每行 ~5-10 字)——同样的内容会 inflate diff_pct 到 100-200%,触发本不该的 v2 重判。
+
+**实测复现**(use clone PR pre-fix):
+- draft markdown 63 行 / ~380 字
+- 拍摄转录 100 行 / 同 ~380 字
+- 内容几乎完全保留(审稿人原句一字不差 + 5 年前反应概念 + 升维段所有金句)
+- 唯一新增:1 句 brand 锚定 "全面拥抱 AI"
+- **line-level diff_pct = 198%** ⚠️ v2 错误触发
+- 语义内容真实 diff ≈ 15-25%
+
+**修复**:拆 metric。
+- **DIFF_METRIC=char_levenshtein_normalized**(新默认)—— [tools/diff_pct.py](tools/diff_pct.py) 先 normalize(去 markdown header / 分隔线 / 列表标记 / 装饰标点 / 折叠所有空白),再算 char-level Levenshtein / max(len_a, len_b)
+- backend 优先级:`rapidfuzz`(C-backed,~ms 级;需 `pip install rapidfuzz`) → `difflib.SequenceMatcher`(stdlib,永远可用,~10ms 级)
+- **V2_TRIGGER_THRESHOLD = 0.30** 保持不动(阈值经验合理)
+- legacy line-level 保留为终极 fallback(只在 python3 + tools/diff_pct.py 都不可达时降级)
+
+**测试**:3 fixture × 2 backend = 6 case,全过:
+
+| Case | 内容 | 期望范围 | difflib | rapidfuzz |
+|---|---|---|---|---|
+| 1 | markdown 长句 vs 转录短断句(内容同) | < 30 | 7 | 12 |
+| 2 | 完全不同主题 | ≥ 60 | 88 | 97 |
+| 3 | 加 20% outro/CTA | 10-30 | 14 | 25 |
+
+跑 `bash tools/diff_pct_test.sh` 复现。
+
+**已知局限**:
+- 历史 v2 prediction 文件保留 line-level 数字作 audit trail——不重新走过去的 prediction
+- normalize 是启发式(中文标点)——对其他语言 / 非典型 markdown 可能需调整 regex
+
### Fixed — `rubric_notes.md` 实绩泄漏漏洞(**BREAKING for blind channel integrity**)
**问题**:PR #11 引入的 cheat-score-blind sub-agent 承诺只读 `scripts/<id>.md` + `rubric_notes.md` 两个文件。但 cheat-bump Phase 5 把升级 Memo(含真实视频名 + 实绩 + 派生证据)写进了 `rubric_notes.md`——sub-agent 通过白名单读到了本不该看的实绩,盲打分变成"看过实绩的事后合理化"。实测复现:5 条已发视频里 2 条 sub-agent 自动标 `any_contamination_signal: true`(refusal=`non_blind_warning`,所有维度 confidence 降 medium)。
diff --git a/skills/cheat-shoot/SKILL.md b/skills/cheat-shoot/SKILL.md
index f0373c5..1a51efb 100644
--- a/skills/cheat-shoot/SKILL.md
+++ b/skills/cheat-shoot/SKILL.md
@@ -41,8 +41,9 @@ cheat-shoot 自己**不**写预测内容——所有预测落盘逻辑在 cheat-
## Constants
- **REQUIRE_PREDICTION = true** — 拍前必须先有 v1 prediction 文件
-- **V2_TRIGGER_THRESHOLD = 0.30** — 稿子 diff 超过 30%(行级 unified diff 行数 / 原稿子行数)→ 默认建议 v2 重判;低于 30% 询问用户是否仍要 v2
-- **DIFF_METRIC = lines** — 用 `diff -u | grep '^[+-]' | wc -l` 算改动行数 / 原文件行数
+- **V2_TRIGGER_THRESHOLD = 0.30** — normalize 后 char-level diff 超过 30% → 默认建议 v2 重判;低于 30% 询问用户是否仍要 v2
+- **DIFF_METRIC = char_levenshtein_normalized**(**默认**)—— 通过 [`tools/diff_pct.py`](../../tools/diff_pct.py) 调用:先 normalize(去 markdown header / 分隔线 / 列表标记 / 装饰标点 / 折叠所有空白),再算 char-level Levenshtein / max(len_a, len_b)。preferred backend `rapidfuzz`,fallback `difflib.SequenceMatcher`(stdlib,永远可用)。**旧版 line-level 在口语化转录场景误报严重**(draft 长 markdown 句 vs whisper 转录的短断句,内容几乎不变但 line-level 算出 ~200% diff)—— PR #14 修复
+- **DIFF_METRIC=lines** —— legacy fallback:当 python3 完全不可用或 tools/diff_pct.py 找不到时降级到 `diff -u | grep '^[+-]' | wc -l` 算法
## Inputs
@@ -99,11 +100,27 @@ c) 大改了,基本是另一条 → 走 _redo 流程:
3. 若用户没保留(即兴)→ 标 `script_lost`,写占位文件 + 警告"v2 重判跳过——下次建议留稿(哪怕 voice memo 转录)",进 Phase 4
4. 提供了的话:算 diff
```bash
- added=$(diff -u scripts/<id>.md videos/<id>/script.md | grep -c '^+')
- removed=$(diff -u scripts/<id>.md videos/<id>/script.md | grep -c '^-')
- total_orig=$(wc -l < scripts/<id>.md)
- diff_pct=$(( (added + removed) * 100 / total_orig ))
+ # 解析 cheat-on-content 源码根(cheat-shoot 是 symlink 装的)
+ SKILL_REAL="$(readlink -f ~/.claude/skills/cheat-shoot 2>/dev/null || readlink ~/.claude/skills/cheat-shoot 2>/dev/null)"
+ if [[ -n "$SKILL_REAL" ]]; then
+ REPO_ROOT="$(cd "$SKILL_REAL/../.." && pwd)"
+ DIFF_TOOL="$REPO_ROOT/tools/diff_pct.py"
+ fi
+
+ if [[ -n "${DIFF_TOOL:-}" && -f "$DIFF_TOOL" ]] && command -v python3 >/dev/null 2>&1; then
+ # 默认 char-level Levenshtein on normalized text(rapidfuzz preferred, difflib fallback)
+ diff_pct=$(python3 "$DIFF_TOOL" "scripts/<id>.md" "videos/<id>/script.md")
+ else
+ # legacy line-level fallback——只在 python3 或 diff_pct.py 都不可用时用
+ added=$(diff -u scripts/<id>.md videos/<id>/script.md | grep -c '^+')
+ removed=$(diff -u scripts/<id>.md videos/<id>/script.md | grep -c '^-')
+ total_orig=$(wc -l < scripts/<id>.md)
+ diff_pct=$(( (added + removed) * 100 / total_orig ))
+ echo "⚠️ fallback 到 line-level diff——口语化转录会 inflate diff_pct,可能误触发 v2"
+ fi
```
+
+ **为什么 normalize + char-level**:line-level diff 在创作者真实场景(draft 是 markdown 长句、拍摄稿是 whisper 转录的口语化短行)算出 ~200% 差异但内容几乎不变。char-level Levenshtein 在 normalize 后稳定反映**内容**差异,而非格式差异。详见 [`tools/diff_pct.py`](../../tools/diff_pct.py) + `tools/diff_pct_test.sh`(3 fixture 在两个 backend 上全过)。
5. **判定 v2 触发**:
- `diff_pct >= 30` → 默认建议 v2 重判,**主动调用** `/cheat-predict — mode: v2 — prediction-file: predictions/<id>.md` 传 `videos/<id>/script.md` 作 input。cheat-predict 走 v2 模式 append `## 预测 v2`
- `diff_pct < 30` → 询问用户:"只改了 N% 的内容,要重判吗?默认不(v1 预测仍有效)"。用户说要 → 同上调用;用户说不 → 跳过 v2,继续 Phase 4
diff --git a/tools/diff_pct.py b/tools/diff_pct.py
new file mode 100755
index 0000000..0a3298b
--- /dev/null
+++ b/tools/diff_pct.py
@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+"""
+diff_pct.py — char-level diff_pct on normalized text.
+
+Replaces the legacy line-level diff in cheat-shoot Phase 3b, which
+inflated diff_pct for spoken-style transcripts (long markdown lines vs
+short transcribed lines) and falsely triggered v2 re-predictions.
+
+Algorithm:
+ 1. Normalize both files (strip markdown elements, collapse whitespace)
+ 2. Char-level Levenshtein distance / max(len_a, len_b)
+ 3. Output integer 0-100
+
+Backend selection:
+ - Prefer rapidfuzz (C-backed, ~ms for 10KB strings) — `pip install rapidfuzz`
+ - Fallback to stdlib difflib.SequenceMatcher (slower but always available)
+
+Usage:
+ python3 tools/diff_pct.py <orig_path> <new_path>
+Output:
+ Single integer 0-100 on stdout.
+
+Exit codes:
+ 0 = success (number on stdout)
+ 2 = bad args
+ 3 = file read error
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+
+# Markdown / formatting noise that whisper-style transcripts don't have.
+# Stripping these from BOTH sides equalizes the surface so we measure
+# content-similarity, not formatting-similarity.
+_MARKDOWN_HEADER = re.compile(r"^#+\s+.*$", re.MULTILINE)
+_MARKDOWN_HR = re.compile(r"^[-=*]{3,}\s*$", re.MULTILINE)
+_MARKDOWN_BLOCKQUOTE = re.compile(r"^>+\s*", re.MULTILINE)
+_MARKDOWN_LIST = re.compile(r"^[-*+]\s+|^\d+\.\s+", re.MULTILINE)
+_FORMATTING_PUNCT = re.compile(r"[「」『』\"`*_~]")
+_ALL_WHITESPACE = re.compile(r"\s+")
+
+
+def normalize(text: str) -> str:
+ """Strip markdown / formatting noise, collapse to single 'sentence'."""
+ text = _MARKDOWN_HEADER.sub("", text)
+ text = _MARKDOWN_HR.sub("", text)
+ text = _MARKDOWN_BLOCKQUOTE.sub("", text)
+ text = _MARKDOWN_LIST.sub("", text)
+ text = _FORMATTING_PUNCT.sub("", text)
+ text = _ALL_WHITESPACE.sub("", text)
+ return text
+
+
+def diff_pct(a: str, b: str) -> tuple[int, str]:
+ """Return (0-100 int, backend_name)."""
+ a_n = normalize(a)
+ b_n = normalize(b)
+ max_len = max(len(a_n), len(b_n))
+ if max_len == 0:
+ return 0, "trivial"
+
+ # Try rapidfuzz first (faster + matches the spec)
+ try:
+ from rapidfuzz.distance import Levenshtein
+
+ d = Levenshtein.distance(a_n, b_n)
+ return (d * 100) // max_len, "rapidfuzz"
+ except ImportError:
+ pass
+
+ # Fallback to stdlib difflib SequenceMatcher.
+ # ratio() returns 0-1 similarity; we want 0-100 distance.
+ # SequenceMatcher's algorithm differs from pure Levenshtein but
+ # for our use case (semantic content similarity) it's well-correlated.
+ from difflib import SequenceMatcher
+
+ sm = SequenceMatcher(a=a_n, b=b_n, autojunk=False)
+ sim = sm.ratio()
+ return int(round((1 - sim) * 100)), "difflib"
+
+
+def main() -> int:
+ if len(sys.argv) != 3:
+ print("usage: diff_pct.py <orig_path> <new_path>", file=sys.stderr)
+ return 2
+
+ try:
+ a = Path(sys.argv[1]).read_text(encoding="utf-8")
+ b = Path(sys.argv[2]).read_text(encoding="utf-8")
+ except OSError as e:
+ print(f"read error: {e}", file=sys.stderr)
+ return 3
+
+ pct, backend = diff_pct(a, b)
+ print(pct)
+ # backend goes to stderr so callers can inspect without parsing stdout
+ print(f"backend={backend} a_norm_len={len(normalize(a))} b_norm_len={len(normalize(b))}", file=sys.stderr)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/diff_pct_test.sh b/tools/diff_pct_test.sh
new file mode 100755
index 0000000..3c2ac61
--- /dev/null
+++ b/tools/diff_pct_test.sh
@@ -0,0 +1,150 @@
+#!/usr/bin/env bash
+#
+# diff_pct_test.sh — regression test for tools/diff_pct.py
+#
+# Validates 3 fixture cases that the legacy line-level diff failed:
+# case 1: long markdown lines vs spoken-transcript short lines, same content
+# → expected diff_pct < 30 (was ~198% under legacy)
+# case 2: completely different topic
+# → expected diff_pct ≥ 60
+# case 3: orig + ~20% new content appended
+# → expected diff_pct 10-30
+#
+# Usage:
+# bash tools/diff_pct_test.sh
+# Exit:
+# 0 = all pass
+# 1 = ≥1 failure
+#
+# Runs against whichever backend is installed (rapidfuzz preferred, difflib
+# fallback). Both should pass these ranges — they're chosen to be wide
+# enough to absorb backend-algorithm differences.
+
+set -euo pipefail
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+DIFF_PCT="$SCRIPT_DIR/diff_pct.py"
+
+if [[ ! -f "$DIFF_PCT" ]]; then
+ echo "❌ diff_pct.py not found at $DIFF_PCT" >&2
+ exit 1
+fi
+
+TMP=$(mktemp -d)
+trap "rm -rf $TMP" EXIT
+
+PASS=0
+FAIL=0
+
+run_case() {
+ local label="$1"
+ local orig="$2"
+ local new="$3"
+ local min="$4"
+ local max="$5"
+
+ local stderr_out
+ stderr_out=$(mktemp)
+ local actual
+ actual=$(python3 "$DIFF_PCT" "$orig" "$new" 2>"$stderr_out")
+ local backend
+ backend=$(grep -oE 'backend=[a-z]+' "$stderr_out" | head -1 || echo "backend=?")
+ rm -f "$stderr_out"
+
+ if (( actual >= min && actual <= max )); then
+ echo " ✓ $label: diff_pct=$actual ∈ [$min, $max] ($backend)"
+ PASS=$((PASS+1))
+ else
+ echo " ✗ $label: diff_pct=$actual NOT in [$min, $max] ($backend)"
+ FAIL=$((FAIL+1))
+ fi
+}
+
+echo ""
+echo "=== Case 1: long markdown line vs spoken-transcript short lines, same content ==="
+
+cat > "$TMP/case1_orig.md" <<'EOF'
+# 视频草稿 — 评审会复盘
+
+「最近发现一个现象」——所有审稿人都在说一样的话:你的研究太老套了。
+
+但你仔细看,他们引用的全是 5 年前的反应。AI 不是新东西。新的是这次大家集体觉醒了。
+
+**升维点**:当所有人都在追新概念的时候,先看穿规律的人已经在用工具赚钱了。
+EOF
+
+cat > "$TMP/case1_shot.md" <<'EOF'
+最近发现一个现象
+所有审稿人都在说一样的话
+你的研究太老套了
+但你仔细看
+他们引用的全是
+5 年前的反应
+AI 不是新东西
+新的是这次
+大家集体觉醒了
+当所有人都在追新概念的时候
+先看穿规律的人
+已经在用工具赚钱了
+EOF
+
+run_case "spoken-style line breaks, content preserved" \
+ "$TMP/case1_orig.md" "$TMP/case1_shot.md" 0 30
+
+echo ""
+echo "=== Case 2: completely different topic ==="
+
+cat > "$TMP/case2_orig.md" <<'EOF'
+# AI 焦虑
+
+最近 AI 大模型发布得太快了,每周一个新工具。
+你刚学会一个,下周它就被淘汰了。
+这种焦虑本质上是工具焦虑,不是能力焦虑。
+真正不变的是你解决问题的范式——选好题、定好评估、跑通闭环。
+EOF
+
+cat > "$TMP/case2_shot.md" <<'EOF'
+今天聊聊我家的猫
+它叫橘子 是一只英短
+最喜欢窝在窗台上晒太阳
+我每天下班回家
+看到它就忘了所有烦恼
+它今天还偷吃了我的牛奶
+被我发现了 装作没事
+猫真是世界上最神奇的生物
+EOF
+
+run_case "completely different topic" \
+ "$TMP/case2_orig.md" "$TMP/case2_shot.md" 60 100
+
+echo ""
+echo "=== Case 3: orig + ~20% appended (outro / CTA scenario) ==="
+
+# Realistic 创作者 scenario: 拍前稿 ~150 字, 拍时加一句 30 字 outro
+# 增量 / 原稿 ≈ 20%, Levenshtein/max(orig,new) ≈ 16-20%
+cat > "$TMP/case3_orig.md" <<'EOF'
+# 视频 — 关于宿命论
+
+我以前不信宿命论。直到我跑了这个工具——它让我拍了一条视频,预测了流量。
+我想证明它是错的,告诉了观众希望集体观测让数据偏移。结果数据是准的。
+我没逃出宿命论。我只是从一阶跳到二阶——AI 在观测观测者。
+EOF
+
+cat > "$TMP/case3_shot.md" <<'EOF'
+我以前不信宿命论。直到我跑了这个工具——它让我拍了一条视频,预测了流量。
+我想证明它是错的,告诉了观众希望集体观测让数据偏移。结果数据是准的。
+我没逃出宿命论。我只是从一阶跳到二阶——AI 在观测观测者。
+此时此刻看到这条的你——是出于好奇,还是在完成算法的最后一次落位?
+EOF
+
+run_case "~20% appended outro" \
+ "$TMP/case3_orig.md" "$TMP/case3_shot.md" 10 30
+
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="
+
+if [[ $FAIL -eq 0 ]]; then
+ exit 0
+else
+ exit 1
+fi