Skip to content

"CLI-Anything: Making ALL Software Agent-Native" -- CLI-Hub: https://clianything.cc/

Python

test: add calibre harness smoke validation

yuhaob2c469244635
diff --git a/calibre/agent-harness/FIX_NOTES.md b/calibre/agent-harness/FIX_NOTES.md
new file mode 100644
index 000000000..c353f17ac
--- /dev/null
+++ b/calibre/agent-harness/FIX_NOTES.md
@@ -0,0 +1,60 @@
+# FIX_NOTES.md - PR #223 Calibre Harness Validation
+
+## Blocker Status
+
+- No code blocker was identified in the Calibre harness.
+- Remaining validation/documentation blocker addressed by adding no-Calibre subprocess smoke coverage and explicit real-backend validation steps.
+
+## No-Calibre Smoke Validation
+
+These commands validate importability, Click entrypoint behavior, and missing-library
+error handling without requiring `calibredb`, `ebook-convert`, or `ebook-meta`.
+
+```bash
+cd calibre/agent-harness
+python -m py_compile \
+ cli_anything/calibre/calibre_cli.py \
+ cli_anything/calibre/core/*.py \
+ cli_anything/calibre/utils/*.py
+python -m pytest cli_anything/calibre/tests/test_core.py -v
+```
+
+To require the installed console script for smoke validation:
+
+```bash
+cd calibre/agent-harness
+pip install -e .
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
+```
+
+## Real Calibre Backend Validation
+
+Install Calibre first and confirm all wrapped commands resolve:
+
+```bash
+which calibredb
+which ebook-convert
+which ebook-meta
+```
+
+Then run the E2E suite:
+
+```bash
+cd calibre/agent-harness
+pip install -e .
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_full_e2e.py -v -s
+```
+
+Expected E2E coverage:
+
+- Temporary Calibre libraries are created for test isolation.
+- Generated EPUB fixtures are imported through `calibredb`.
+- Metadata changes are round-tripped through the real backend.
+- EPUB to TXT/MOBI conversion runs through `ebook-convert`.
+- Exported and converted artifacts are checked for existence and nonzero size.
+
+## Remaining Gaps
+
+- Catalog generation, file-level metadata embedding, and real custom-column workflows are documented as E2E gaps in `cli_anything/calibre/tests/TEST.md`.
diff --git a/calibre/agent-harness/cli_anything/calibre/README.md b/calibre/agent-harness/cli_anything/calibre/README.md
index d102a5606..2e84d545e 100644
--- a/calibre/agent-harness/cli_anything/calibre/README.md
+++ b/calibre/agent-harness/cli_anything/calibre/README.md
@@ -177,12 +177,38 @@ Override with the `CALIBRE_LIBRARY` environment variable.
```bash
cd agent-harness
-# Unit tests (no Calibre required)
-pytest cli_anything/calibre/tests/test_core.py -v
+# Syntax check
+python -m py_compile \
+ cli_anything/calibre/calibre_cli.py \
+ cli_anything/calibre/core/*.py \
+ cli_anything/calibre/utils/*.py
+
+# Unit and CLI smoke tests (no Calibre required)
+python -m pytest cli_anything/calibre/tests/test_core.py -v
+
+# Installed-command smoke tests (no Calibre required)
+pip install -e .
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
# Full E2E tests (Calibre required)
-pytest cli_anything/calibre/tests/test_full_e2e.py -v -s
+python -m pytest cli_anything/calibre/tests/test_full_e2e.py -v -s
-# All tests with installed binary
-CLI_ANYTHING_FORCE_INSTALLED=1 pytest cli_anything/calibre/tests/ -v -s
+# All tests with installed binary and real Calibre backend
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest cli_anything/calibre/tests/ -v -s
```
+
+### Real Backend Validation
+
+Before running E2E validation, verify Calibre's commands are on PATH:
+
+```bash
+which calibredb
+which ebook-convert
+which ebook-meta
+```
+
+The E2E suite creates temporary Calibre libraries, imports generated EPUB files,
+updates metadata with `calibredb`, converts EPUB files with `ebook-convert`, and
+checks exported/converted artifacts for real output files. These tests require a
+real Calibre installation; `test_core.py` remains the no-Calibre validation path.
diff --git a/calibre/agent-harness/cli_anything/calibre/tests/TEST.md b/calibre/agent-harness/cli_anything/calibre/tests/TEST.md
index 8487e4425..92c4e7a67 100644
--- a/calibre/agent-harness/cli_anything/calibre/tests/TEST.md
+++ b/calibre/agent-harness/cli_anything/calibre/tests/TEST.md
@@ -5,7 +5,13 @@
This document covers the test plan and results for `cli-anything-calibre`, a CLI harness
wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
-**Hard dependency:** Calibre must be installed (`calibredb` in PATH).
+**Unit/smoke dependency:** `test_core.py` does not require Calibre. It includes
+subprocess smoke checks for help, version, and missing-library behavior using the
+installed `cli-anything-calibre` command when available, or `python -m
+cli_anything.calibre` as a development fallback.
+
+**E2E hard dependency:** `test_full_e2e.py` requires Calibre (`calibredb`,
+`ebook-convert`, and `ebook-meta` in PATH) for real backend validation.
---
@@ -13,8 +19,8 @@ wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
| File | Tests Planned | Description |
|------|--------------|-------------|
-| `test_core.py` | 38 | Unit tests: synthetic data, no external deps, no Calibre needed |
-| `test_full_e2e.py` | 20 | E2E tests: real Calibre library operations + subprocess CLI tests |
+| `test_core.py` | 41 | Unit tests: synthetic data, no external deps, no Calibre needed, plus subprocess smoke |
+| `test_full_e2e.py` | 21 | E2E tests: real Calibre library operations + subprocess CLI tests |
---
@@ -72,13 +78,36 @@ wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
| `test_find_ebook_meta_missing` | find_ebook_meta() raises RuntimeError if not in PATH |
| `test_error_message_contains_install_hint` | Error messages include install instructions |
-### `calibre_cli.py` — 3 tests
+### `calibre_cli.py` — 6 tests
| Test | Description |
|------|-------------|
| `test_cli_help_exits_zero` | `--help` prints usage and returns 0 |
| `test_cli_version` | `--version` returns version string |
| `test_cli_missing_library_error` | Commands without library set print clear error |
+| `test_installed_or_module_help_smoke` | Subprocess `--help` smoke test; uses installed entry point if present |
+| `test_installed_or_module_version_smoke` | Subprocess `--version` smoke test; uses installed entry point if present |
+| `test_missing_library_error_without_calibre` | Subprocess missing-library error works without Calibre installed |
+
+Run no-backend validation:
+
+```bash
+cd calibre/agent-harness
+python -m py_compile \
+ cli_anything/calibre/calibre_cli.py \
+ cli_anything/calibre/core/*.py \
+ cli_anything/calibre/utils/*.py
+python -m pytest cli_anything/calibre/tests/test_core.py -v
+```
+
+Installed-command smoke mode:
+
+```bash
+cd calibre/agent-harness
+pip install -e .
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
+```
---
@@ -91,14 +120,15 @@ These tests invoke the **real Calibre** tools and verify the output.
E2E tests use a temporary Calibre library created with `calibredb add` from a real EPUB.
A minimal EPUB is generated programmatically (valid ZIP structure) for reproducibility.
-### `TestLibraryOperations` — 5 tests
+### `TestLibraryOperations` — 6 tests
| Test | Description | Verified |
|------|-------------|---------|
-| `test_create_library_and_add_epub` | Add EPUB to new library | Book ID returned, book count > 0 |
+| fixture setup | Add EPUB to new library | Book ID returned, book count > 0 |
| `test_list_books` | list_books() returns book entries | ID, title, authors present |
+| `test_list_books_custom_fields` | list_books() honors explicit field list | Requested fields present, omitted fields absent |
| `test_search_books` | search_books() with query | Returns matching IDs |
-| `test_show_metadata` | get_metadata() returns parsed OPF | title, authors fields present |
+| `test_get_metadata` | get_metadata() returns parsed OPF | title, authors fields present |
| `test_export_books` | export_books() exports to directory | Files exported, cover.jpg present |
### `TestMetadataOperations` — 3 tests
@@ -182,7 +212,39 @@ Tests the installed `cli-anything-calibre` command directly via subprocess.
## Test Results
-Run command:
+No-backend validation run:
+
+```bash
+cd calibre/agent-harness
+python -m py_compile \
+ cli_anything/calibre/calibre_cli.py \
+ cli_anything/calibre/core/*.py \
+ cli_anything/calibre/utils/*.py
+python -m pytest cli_anything/calibre/tests/test_core.py -v
+```
+
+Current no-backend result:
+
+```text
+41 passed in 0.74s
+```
+
+Installed-command smoke run:
+
+```bash
+cd calibre/agent-harness
+pip install -e .
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
+```
+
+Current installed-command smoke result:
+
+```text
+3 passed in 0.63s
+```
+
+Historical real-backend run:
```bash
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest cli_anything/calibre/tests/ -v --tb=no
```
@@ -250,14 +312,38 @@ cli_anything/calibre/tests/test_full_e2e.py::TestCLISubprocess::test_version PAS
| Metric | Value |
|--------|-------|
-| Total tests | 50 |
-| Passed | 50 |
+| Total tests expected now | 62 |
+| No-backend tests passed in current run | 41 |
+| Installed smoke tests passed in current run | 3 |
+| Historical full-suite tests passed | 50 |
| Failed | 0 |
-| Pass rate | 100% |
-| Execution time | 15.93s |
+| Current no-backend execution time | 0.74s |
+| Current installed-smoke execution time | 0.63s |
+| Historical full-suite execution time | 15.93s |
| Calibre version | 7.6 |
| Subprocess backend | `/home/orgleaf/py-base-venv/bin/cli-anything-calibre` (installed) |
+## Real Backend Validation Steps
+
+Use these steps to validate the harness against a real Calibre install:
+
+```bash
+cd calibre/agent-harness
+pip install -e .
+which calibredb
+which ebook-convert
+which ebook-meta
+CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
+ cli_anything/calibre/tests/test_full_e2e.py -v -s
+```
+
+Expected behavior:
+
+- Tests create temporary Calibre libraries and seed them with generated EPUB files.
+- `calibredb` is used for add/list/search/metadata/export operations.
+- `ebook-convert` is used for EPUB to TXT/MOBI conversion and output files are checked for existence and nonzero size.
+- Subprocess E2E tests require the installed `cli-anything-calibre` entry point when `CLI_ANYTHING_FORCE_INSTALLED=1` is set.
+
## Coverage Notes
- All session persistence operations are tested (save, load, env override, error paths)
diff --git a/calibre/agent-harness/cli_anything/calibre/tests/test_core.py b/calibre/agent-harness/cli_anything/calibre/tests/test_core.py
index 1cc40b1c2..69672669e 100644
--- a/calibre/agent-harness/cli_anything/calibre/tests/test_core.py
+++ b/calibre/agent-harness/cli_anything/calibre/tests/test_core.py
@@ -5,6 +5,8 @@ All tests use synthetic data — no real Calibre installation required.
import json
import os
+import shutil
+import subprocess
import sys
import tempfile
import unittest
@@ -614,5 +616,59 @@ class TestCLIHelp(unittest.TestCase):
)
+class TestCLISubprocessSmoke(unittest.TestCase):
+ """Subprocess smoke tests that do not require Calibre to be installed."""
+
+ @staticmethod
+ def _resolve_cli():
+ force = os.environ.get("CLI_ANYTHING_FORCE_INSTALLED", "").strip() == "1"
+ installed = shutil.which("cli-anything-calibre")
+ if installed:
+ return [installed]
+ if force:
+ raise RuntimeError(
+ "cli-anything-calibre not found in PATH. Install with: pip install -e ."
+ )
+ return [sys.executable, "-m", "cli_anything.calibre"]
+
+ def _run(self, args, *, home=None):
+ env = os.environ.copy()
+ harness_root = Path(__file__).resolve().parents[3]
+ env["PYTHONPATH"] = (
+ str(harness_root)
+ if not env.get("PYTHONPATH")
+ else f"{harness_root}{os.pathsep}{env['PYTHONPATH']}"
+ )
+ env.pop("CALIBRE_LIBRARY", None)
+ if home:
+ env["HOME"] = home
+ return subprocess.run(
+ self._resolve_cli() + args,
+ capture_output=True,
+ text=True,
+ env=env,
+ check=False,
+ )
+
+ def test_installed_or_module_help_smoke(self):
+ result = self._run(["--help"])
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertIn("Usage:", result.stdout)
+ self.assertIn("library", result.stdout)
+
+ def test_installed_or_module_version_smoke(self):
+ result = self._run(["--version"])
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertIn("cli-anything-calibre", result.stdout)
+
+ def test_missing_library_error_without_calibre(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ result = self._run(["books", "list"], home=tmp)
+ combined = result.stdout + result.stderr
+ self.assertNotEqual(result.returncode, 0)
+ self.assertIn("No Calibre library connected", combined)
+ self.assertIn("CALIBRE_LIBRARY", combined)
+
+
if __name__ == "__main__":
unittest.main()