Turn any technical book PDF into a Claude Code skill — ready to study, reference, and use while you work.
Python agent-skillsai-agentsbook-to-skillcontext-engineeringdocument-processingedtechknowledge-baseknowledge-managementllmpdf-to-markdownragself-studystudy-tools
fix(config): give each run its own workdir so concurrent extractions cannot clobber each other (#184)
Jean Giet7bcfcd526232
diff --git a/SKILL.md b/SKILL.mdindex bb860ba..12e33b2 100644--- a/SKILL.md+++ b/SKILL.md@@ -163,17 +163,21 @@ Before extraction, the script checks optional Python packages needed for the det**Tip — preflight the environment:** run `"$PYTHON_BIN" "$SCRIPT_PATH" --check` to print a per-format report of which extractors are installed and the exact command to install whatever is missing, without processing any file. Useful when a user reports a setup or quality problem.-This creates:-- `<tempdir>/book_skill_work/full_text.txt` — combined extracted text of all sources with clear visually demarcated boundaries.-- `<tempdir>/book_skill_work/metadata.json` — overall combined size, words, pages, token counts, dropped EPUB image counts, and a detailed list of individual processed `sources`.+This creates a **per-run** work directory — `<tempdir>/book_skill_work-<pid>/` by default, or exactly the path you set in `BOOK_SKILL_WORKDIR` — containing:+- `full_text.txt` — combined extracted text of all sources with clear visually demarcated boundaries.+- `metadata.json` — overall combined size, words, pages, token counts, dropped EPUB image counts, the resolved `workdir`, and a detailed list of individual processed `sources`.-Read `<tempdir>/book_skill_work/metadata.json` to inspect the results.+The run prints all three paths on completion (`Workdir ->`, `Text ->`, `Meta ->`). **Take the paths from that output (or from `metadata.json`'s own `workdir` field) rather than assuming a fixed location** — the directory name differs per run so that concurrent extractions on one machine cannot overwrite each other's results.++Read that run's `metadata.json` to inspect the results.++**Always confirm the extraction is the document you asked for** before generating anything: check `filename` / `source_file` in `metadata.json`, or the `SOURCE:` header on the first line of `full_text.txt`. If you are waiting on a background run, wait on *its* specific workdir — polling a shared path can surface a different run's output.---## Step 2.5 — Pre-flight cost estimate-Read `<tempdir>/book_skill_work/metadata.json` and present the user with an estimate **before doing any generation**:+Read this run's `metadata.json` (the `Meta ->` path from the extraction output) and present the user with an estimate **before doing any generation**:```📖 Sources detected: <total_sources> source(s)@@ -542,18 +546,35 @@ if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; thenPYTHON_BIN="python"fi-"$PYTHON_BIN" - <<'PY'-import os+Remove **the work directory this run actually used** — the `Workdir ->` path from the+extraction output, which is also stored as `workdir` in `metadata.json`. Never delete a+directory you did not create: another extraction may be running beside yours.++```bash+# WORKDIR is the path this run reported; quote it in case of spaces.+rm -rf "$WORKDIR"+```++Equivalently, if you still have the metadata file:++```bash+"$PYTHON_BIN" - "$WORKDIR_METADATA_JSON" <<'PY'+import jsonimport shutil-import tempfile+import sysfrom pathlib import Path-shutil.rmtree(- os.environ.get("BOOK_SKILL_WORKDIR", Path(tempfile.gettempdir()) / "book_skill_work"),- ignore_errors=True,-)++meta_path = Path(sys.argv[1])+workdir = json.loads(meta_path.read_text(encoding="utf-8")).get("workdir")+if workdir:+ shutil.rmtree(workdir, ignore_errors=True)PY```+Older copies of this file removed a single fixed `book_skill_work` directory. That path is+no longer used, so such a cleanup is now a harmless no-op rather than something that could+delete a concurrent run's output.+Then report to the user:```@@ -653,7 +674,7 @@ Read and parse the existing skill's files:- Read `$SKILLS_HOME/<skill_name>/glossary.md`, `$SKILLS_HOME/<skill_name>/patterns.md`, and `$SKILLS_HOME/<skill_name>/cheatsheet.md` to see what terms and frameworks are already indexed.### 2. Match Content & Identify Revisions vs. Additions-Analyze the new extracted text in `<tempdir>/book_skill_work/full_text.txt` to identify if the new content represents:+Analyze the new extracted text in this run's `full_text.txt` (the `Text ->` path from the extraction output) to identify if the new content represents:- **Updates/Revisions to existing chapters**: If a section of the new content directly updates or expands an existing chapter's topic, read the existing chapter file, merge the new details into it, and rewrite the file.- **New additions**: If the content introduces new chapters, papers, or separate sections, create **new chapter summary files** under `chapters/`. Start numbering these files after the highest existing chapter number (e.g. if the existing chapters stop at `ch12`, create `ch13-*.md`, `ch14-*.md`, etc.).diff --git a/book_to_skill/config.py b/book_to_skill/config.pyindex 18ca0ce..b33130f 100644--- a/book_to_skill/config.py+++ b/book_to_skill/config.py@@ -2,12 +2,29 @@ import osimport tempfilefrom pathlib import Path-OUTPUT_DIR = Path(- os.environ.get(- "BOOK_SKILL_WORKDIR",- str(Path(tempfile.gettempdir()) / "book_skill_work"),- )-)+def default_output_dir() -> Path:+ """Per-run work directory, unique to this process.++ The PID is part of the name so two extractions running at the same time+ cannot overwrite each other. Every run previously shared one fixed path+ ($TMPDIR/book_skill_work), so whichever run finished second silently+ replaced the first run's full_text.txt and metadata.json — and an agent+ polling for metadata.json could pick up a *different document's*+ extraction without any error, then build a skill from the wrong source.++ The name is deliberately a sibling of the old fixed path rather than a+ child of it. An older cleanup routine that removes "book_skill_work"+ then simply finds nothing, instead of deleting a live concurrent run.++ BOOK_SKILL_WORKDIR still overrides this completely.+ """+ return Path(tempfile.gettempdir()) / f"book_skill_work-{os.getpid()}"+++# `or` rather than a get() default: BOOK_SKILL_WORKDIR set to an empty string+# would otherwise become Path(""), i.e. the current directory — which the run+# would then populate and chmod to 0700.+OUTPUT_DIR = Path(os.environ.get("BOOK_SKILL_WORKDIR") or default_output_dir())OUTPUT_TEXT = OUTPUT_DIR / "full_text.txt"OUTPUT_META = OUTPUT_DIR / "metadata.json"diff --git a/book_to_skill/utils.py b/book_to_skill/utils.pyindex f0eb504..2b2d7ee 100644--- a/book_to_skill/utils.py+++ b/book_to_skill/utils.py@@ -1066,6 +1066,9 @@ def main():"estimated_tokens": total_tokens,"estimated_tokens_human": f"~{total_tokens // 1000}K","images_dropped": total_images_dropped,+ # Self-describing so a consumer can clean up exactly the directory this+ # run created, without having to reconstruct the per-run default path.+ "workdir": str(OUTPUT_DIR),"output_text": str(OUTPUT_TEXT),"total_sources": len(extracted_sources),"sources": [@@ -1127,8 +1130,9 @@ def main():" WARN : No table of contents detected — chapter mapping in Step 3 ""will rely on heading scan only, which may miss or duplicate sections.")- print(f"\n Text -> {OUTPUT_TEXT}")- print(f" Meta -> {OUTPUT_META}")+ print(f"\n Workdir -> {OUTPUT_DIR}")+ print(f" Text -> {OUTPUT_TEXT}")+ print(f" Meta -> {OUTPUT_META}")if errors:print(f"\n WARNING: {len(errors)} source(s) skipped due to errors:")for path, err in errors:diff --git a/tests/test_per_run_workdir.py b/tests/test_per_run_workdir.pynew file mode 100644index 0000000..ae3cf70--- /dev/null+++ b/tests/test_per_run_workdir.py@@ -0,0 +1,99 @@+"""Regression tests: each run gets its own work directory.++Every extraction used to write to one fixed path ($TMPDIR/book_skill_work),+so two runs started at the same time — routine when several agent sessions+share a machine — silently overwrote each other's full_text.txt and+metadata.json. Nothing errored: the run that finished second simply replaced+the first one's output, and an agent waiting on metadata.json could pick up a+*different document's* extraction and build a skill from the wrong source.++The default now carries the PID. BOOK_SKILL_WORKDIR still overrides it, and an+empty BOOK_SKILL_WORKDIR must not collapse to Path("") — i.e. the current+directory, which the run would then populate and chmod to 0700.+"""++import importlib+import os+import sys+import tempfile+from pathlib import Path++ROOT_DIR = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(ROOT_DIR))++from book_to_skill.config import default_output_dir # noqa: E402+++def _reimport_config(monkeypatch, workdir_env):+ """Re-evaluate config.py's module-level OUTPUT_DIR under a given env."""+ if workdir_env is None:+ monkeypatch.delenv("BOOK_SKILL_WORKDIR", raising=False)+ else:+ monkeypatch.setenv("BOOK_SKILL_WORKDIR", workdir_env)+ import book_to_skill.config as config++ return importlib.reload(config)+++def test_default_workdir_is_unique_per_process(monkeypatch):+ """Two concurrent runs must not share a directory."""+ monkeypatch.setattr(os, "getpid", lambda: 1111)+ first = default_output_dir()+ monkeypatch.setattr(os, "getpid", lambda: 2222)+ second = default_output_dir()++ assert first != second+ assert "1111" in first.name+ assert "2222" in second.name+++def test_default_workdir_is_sibling_not_child_of_legacy_path(monkeypatch):+ """An older cleanup that removes "book_skill_work" must not hit a live run.++ Nesting under the legacy directory would mean a stale cleanup routine+ deleting a concurrent extraction's output, which is worse than the bug+ being fixed. The per-run directory is therefore a sibling.+ """+ monkeypatch.setattr(os, "getpid", lambda: 4242)+ legacy = Path(tempfile.gettempdir()) / "book_skill_work"++ assert legacy not in default_output_dir().parents+++def test_default_workdir_lives_in_tempdir(monkeypatch):+ monkeypatch.setattr(os, "getpid", lambda: 4242)++ assert default_output_dir().parent == Path(tempfile.gettempdir())+++def test_env_override_wins(monkeypatch, tmp_path):+ chosen = tmp_path / "private-workdir"+ config = _reimport_config(monkeypatch, str(chosen))++ assert config.OUTPUT_DIR == chosen+ assert config.OUTPUT_TEXT == chosen / "full_text.txt"+ assert config.OUTPUT_META == chosen / "metadata.json"+++def test_empty_env_override_does_not_become_cwd(monkeypatch):+ """BOOK_SKILL_WORKDIR="" must fall back, not resolve to Path("")."""+ config = _reimport_config(monkeypatch, "")++ assert config.OUTPUT_DIR != Path("")+ assert config.OUTPUT_DIR != Path.cwd()+ assert config.OUTPUT_DIR.parent == Path(tempfile.gettempdir())+++def test_unset_env_uses_per_run_default(monkeypatch):+ config = _reimport_config(monkeypatch, None)++ assert config.OUTPUT_DIR.name.startswith("book_skill_work-")+ assert str(os.getpid()) in config.OUTPUT_DIR.name+++def test_config_restored_for_other_tests(monkeypatch):+ """Reloading config in this module must not leak into the rest of the suite."""+ config = _reimport_config(monkeypatch, None)++ assert config.OUTPUT_TEXT.parent == config.OUTPUT_DIR+ assert config.OUTPUT_META.parent == config.OUTPUT_DIR