LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
[FIX] Fix SharePoint walk() not supporting detail=True (#1843)
Chandrasekharan M9e84d45511a3
diff --git a/.claude/skills/connector-ops/SKILL.md b/.claude/skills/connector-ops/SKILL.mdindex 2cbc34c25..56a315586 100644--- a/.claude/skills/connector-ops/SKILL.md+++ b/.claude/skills/connector-ops/SKILL.md@@ -66,11 +66,12 @@ Ask the user for:Use web search to discover:1. **Official Python library** for the service-2. **All supported authentication modes** (API key, OAuth, connection string, certificates, etc.)-3. **Required connection parameters** for each auth mode-4. **Official logo/icon** source+2. **Existing fsspec provider** — check if a mature fsspec-compatible library exists (e.g., `s3fs` for S3, `adlfs` for Azure, `gcsfs` for GCS, `boxfs` for Box). **Always prefer wrapping an existing provider over building a custom one.**+3. **All supported authentication modes** (API key, OAuth, connection string, certificates, etc.)+4. **Required connection parameters** for each auth mode+5. **Official logo/icon** source-Document findings before proceeding.+Document findings before proceeding. If no fsspec provider exists, note that a custom `AbstractFileSystem` subclass will be needed (see Step 5b).### Step 3: Generate Connector ID@@ -124,10 +125,33 @@ Read the template and adapt it for the specific service. Key methods to implemen- `execute()` → Execute queries (inherited, may override)**For Filesystem Connectors:**-- `get_fsspec_fs()` → Return fsspec filesystem instance++There are two approaches depending on whether an fsspec provider exists:++**Approach 1: Wrapping an existing fsspec provider (PREFERRED)**+When a library like `s3fs`, `adlfs`, `gcsfs`, `boxfs` exists, the connector only extends `UnstractFileSystem`:+- `get_fsspec_fs()` → Return the provider's filesystem instance (e.g., `S3FileSystem(...)`)- `test_credentials()` → Verify connection works- `extract_metadata_file_hash()` → Extract file hash from metadata- `is_dir_by_metadata()` → Check if path is directory+- `extract_modified_date()` → Extract last modified date+- **Do NOT implement `ls`, `walk`, `info`, etc.** — the provider handles all file operations.+- See `minio.py` for reference.++**Approach 2: Custom AbstractFileSystem (ONLY when no provider exists)**+Create a custom `AbstractFileSystem` subclass with ONLY these core methods:+- `ls(path, detail=True)` — List directory contents (REQUIRED)+- `info(path)` — Get single item metadata (optimization, optional)+- `_open(path, mode)` — Open a file for reading+- `cat_file(path)` — Read file contents+- `pipe_file(path, value)` — Write bytes (NOT `write_bytes` — that's an alias)+- `rm(path)` — Delete a file+- `mkdir(path)` — Create directory++**CRITICAL: NEVER override** `listdir()`, `stat()`, `exists()`, `isdir()`, `isfile()`, `delete()`, `read_bytes()`, or `write_bytes()`. These are provided by `AbstractFileSystem` and delegate to `ls()`/`info()`/`pipe_file()`.++**`walk()` exception**: Override `walk()` ONLY if the filesystem has path normalization that differs from `_strip_protocol` (e.g., SharePoint maps `""` to `"root"`), or if the service raises exceptions that aren't `FileNotFoundError`/`OSError` (the only types the base catches). If you override `walk()`, you MUST support the `detail` kwarg (return dicts when `True`, lists when `False`) and the `on_error` callback.+- See `sharepoint.py` for reference.**For Queue Connectors:**- `get_engine()` → Return queue connection@@ -467,3 +491,7 @@ Use these templates as starting points:4. **Icon Naming**: Use PascalCase with spaces URL-encoded (e.g., `Google%20Drive.png`).5. **Test Isolation**: Mock tests should never require external services. Use `@unittest.skipUnless` for integration tests.++6. **fsspec Delegation**: NEVER override `walk()`, `listdir()`, `stat()`, `exists()`, `isdir()`, `isfile()`, `delete()`, `read_bytes()`, or `write_bytes()` in custom `AbstractFileSystem` subclasses. These are provided by fsspec's base class and delegate to `ls()`/`info()`/`pipe_file()`. Overriding them creates maintenance burden and introduces subtle bugs. See `references/connector_patterns.md` for the "Anti-Pattern: Reimplementing fsspec Base Methods" section.++7. **pipe_file vs write_bytes**: When implementing file writes in a custom `AbstractFileSystem`, override `pipe_file()` (not `write_bytes()`). The base `write_bytes()` is an alias that delegates to `pipe_file()`.diff --git a/.claude/skills/connector-ops/assets/templates/filesystem.py.template b/.claude/skills/connector-ops/assets/templates/filesystem.py.templateindex 882e23f97..83d0fdfeb 100644--- a/.claude/skills/connector-ops/assets/templates/filesystem.py.template+++ b/.claude/skills/connector-ops/assets/templates/filesystem.py.template@@ -1,6 +1,47 @@"""Filesystem Connector Template+There are TWO approaches for filesystem connectors:++## Approach 1: Wrapping an existing fsspec provider (PREFERRED)+Use this when a mature fsspec-compatible library exists (e.g., s3fs, adlfs, gcsfs, boxfs).+The connector class extends UnstractFileSystem and delegates all file operations to the+provider via get_fsspec_fs(). You do NOT implement ls, walk, info, etc. — the provider+handles all of that.++## Approach 2: Custom AbstractFileSystem (RARE)+Use this ONLY when no fsspec provider exists for the service. You must create a custom+AbstractFileSystem subclass implementing ONLY these core methods:+ - ls(path, detail=True) — List directory contents (REQUIRED)+ - info(path) — Get single item metadata (optimization, optional)+ - _open(path, mode) — Open a file for reading (REQUIRED for reads)+ - cat_file(path) — Read file contents (REQUIRED for reads)+ - pipe_file(path, value) — Write bytes to file (REQUIRED for writes)+ - rm(path) — Delete a file (REQUIRED for writes)+ - mkdir(path) — Create directory (REQUIRED for writes)++NEVER override these methods — AbstractFileSystem already provides them by delegating+to ls()/info():+ - walk() — delegates to ls(), handles detail=True, on_error, topdown.+ EXCEPTION: override walk() if the filesystem has custom path+ normalization (e.g., "" → "root") or raises exceptions that aren't+ FileNotFoundError/OSError. If overriding, MUST support detail kwarg.+ - listdir() — alias for ls()+ - stat() — alias for info()+ - exists() — calls info(), catches exceptions+ - isdir() — calls info(), checks type+ - isfile() — calls info(), checks type+ - delete() — alias for rm()+ - read_bytes()— alias for cat_file()+ - write_bytes() — alias for pipe_file()++Overriding these creates maintenance burden and risks bugs (e.g., walk() not+supporting detail=True, which breaks file discovery).++Reference existing connectors:+ - Approach 1: See minio.py (wraps s3fs.S3FileSystem)+ - Approach 2: See sharepoint.py (custom SharePointFileSystem)+Replace placeholders:- {ClassName}: PascalCase class name (e.g., MinioFS, AzureBlobFS)- {connector_name}: lowercase connector name (e.g., minio, azure_blob)@@ -90,8 +131,7 @@ class {ClassName}(UnstractFileSystem):return ""def get_fsspec_fs(self) -> AbstractFileSystem:- """- Return fsspec filesystem instance.+ """Return fsspec filesystem instance.Uses lazy initialization with thread safety for Celery compatibility.@@ -121,8 +161,7 @@ class {ClassName}(UnstractFileSystem):return self._fsdef test_credentials(self) -> bool:- """- Test filesystem credentials.+ """Test filesystem credentials.Returns:True if connection successful@@ -142,8 +181,7 @@ class {ClassName}(UnstractFileSystem):) from edef extract_metadata_file_hash(self, metadata: dict[str, Any]) -> str | None:- """- Extract unique file hash from fsspec metadata.+ """Extract unique file hash from fsspec metadata.Different storage systems use different keys for file hashes:- S3/Minio: ETag@@ -170,8 +208,7 @@ class {ClassName}(UnstractFileSystem):return Nonedef is_dir_by_metadata(self, metadata: dict[str, Any]) -> bool:- """- Check if path is a directory from metadata.+ """Check if path is a directory from metadata.Args:metadata: File metadata from fsspecdiff --git a/.claude/skills/connector-ops/references/connector_patterns.md b/.claude/skills/connector-ops/references/connector_patterns.mdindex a606d67d4..2d8fa4f0b 100644--- a/.claude/skills/connector-ops/references/connector_patterns.md+++ b/.claude/skills/connector-ops/references/connector_patterns.md@@ -207,6 +207,82 @@ class MyStorageFS(UnstractFileSystem):---+## Anti-Pattern: Reimplementing fsspec Base Methods++**Problem**: When creating a custom `AbstractFileSystem` subclass (no existing fsspec+provider), it's tempting to implement every method for "completeness". This creates+maintenance burden and introduces bugs when the calling code evolves.++**Real-world bug**: SharePoint connector overrode `walk()` but didn't support the+`detail=True` parameter. When file discovery code switched to `walk(detail=True)`,+SharePoint returned plain lists instead of dicts, breaking file processing silently.++**Bad** — reimplementing methods the base class already provides:+```python+class MyFileSystem(AbstractFileSystem):+ def ls(self, path, detail=True, **kwargs):+ ... # Core method - MUST implement++ # BAD: These are all redundant reimplementations+ def listdir(self, path, detail=True, **kwargs):+ return self.ls(path, detail=detail, **kwargs) # Base already does this++ def stat(self, path, **kwargs):+ return self.info(path, **kwargs) # Base already does this++ def exists(self, path, **kwargs):+ try:+ self.info(path)+ return True+ except Exception:+ return False # Base already does this++ def isdir(self, path):+ return self.info(path)["type"] == "directory" # Base already does this++ def walk(self, path, maxdepth=None, **kwargs):+ ... # Base already delegates to ls() with full detail/on_error support++ def delete(self, path, **kwargs):+ self.rm(path, **kwargs) # Base already does this++ def read_bytes(self, path):+ return self.cat_file(path) # Base already does this++ def write_bytes(self, path, data, **kwargs):+ ... # Base already delegates to pipe_file()+```++**Good** — implement only the core methods, let fsspec handle the rest:+```python+class MyFileSystem(AbstractFileSystem):+ # REQUIRED: Core methods that talk to the service API+ def ls(self, path, detail=True, **kwargs): ...+ def info(self, path, **kwargs): ... # Optional optimization+ def _open(self, path, mode="rb", **kwargs): ...+ def cat_file(self, path, **kwargs): ...+ def pipe_file(self, path, value, **kwargs): ... # NOT write_bytes+ def rm(self, path, recursive=False, **kwargs): ...+ def mkdir(self, path, create_parents=True, **kwargs): ...++ # Everything else (walk, exists, isdir, isfile, listdir, stat,+ # delete, read_bytes, write_bytes) is inherited from AbstractFileSystem+```++**Key insight**: `write_bytes` is an alias for `pipe_file` in fsspec. Override+`pipe_file`, not `write_bytes`, so the delegation chain works correctly.++**Exception: `walk()` may need a custom override** when the custom filesystem has+path normalization that differs from `_strip_protocol` (e.g., SharePoint maps `""`+to `"root"` for the Graph API), or when the service raises exceptions that aren't+`FileNotFoundError`/`OSError` (the only types the base `walk()` catches). If you+override `walk()`, you MUST support the `detail` kwarg (return dicts when True,+lists when False) and the `on_error` callback.++**Applies to**: Any custom `AbstractFileSystem` subclass (SharePoint, OneDrive, etc.)++---+## Anti-Pattern: Module-Level Heavy Imports**Bad**:diff --git a/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py b/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.pyindex 9b39c67c8..7491fbf7a 100644--- a/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py+++ b/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py@@ -11,7 +11,6 @@ Uses Microsoft Graph API via Office365-REST-Python-Client library.import loggingimport osimport threading-from collections.abc import Callablefrom datetime import UTC, datetimefrom typing import Any, NoReturn@@ -50,8 +49,9 @@ class SharePointFile(AbstractBufferedFile):return self._file_content[start:end]def _upload_chunk(self, final: bool = False) -> bool:- """Upload is handled by write_bytes."""- return True+ raise NotImplementedError(+ "Buffered writes not supported; use pipe_file()/write_bytes()."+ )class SharePointFileSystem(AbstractFileSystem):@@ -243,10 +243,6 @@ class SharePointFileSystem(AbstractFileSystem):logger.error(f"Error listing path {path}: {e}")raise- def listdir(self, path: str = "", detail: bool = True, **kwargs: Any) -> list[Any]:- """List directory contents (alias for ls)."""- return self.ls(path, detail=detail, **kwargs)-def _item_to_info(self, item: Any, parent_path: str = "") -> dict[str, Any]:"""Convert SharePoint item to fsspec info dict."""name = item.name@@ -308,34 +304,6 @@ class SharePointFileSystem(AbstractFileSystem):parent = "/".join(path.strip("/").split("/")[:-1])return self._item_to_info(item, parent)- def stat(self, path: str, **kwargs: Any) -> dict[str, Any]:- """Get file/directory stats (alias for info)."""- return self.info(path, **kwargs)-- def exists(self, path: str, **kwargs: Any) -> bool:- """Check if path exists."""- try:- self.info(path)- return True- except Exception:- return False-- def isdir(self, path: str) -> bool:- """Check if path is a directory."""- try:- info = self.info(path)- return info.get("type") == "directory"- except Exception:- return False-- def isfile(self, path: str) -> bool:- """Check if path is a file."""- try:- info = self.info(path)- return info.get("type") == "file"- except Exception:- return False-def mkdir(self, path: str, create_parents: bool = True, **kwargs: Any) -> None:"""Create directory."""path = self._normalize_path(path)@@ -382,73 +350,37 @@ class SharePointFileSystem(AbstractFileSystem):mode: str = "rb",**kwargs: Any,) -> SharePointFile:- """Open a file."""+ """Open a file for reading. Write modes are not supported."""+ if any(flag in mode for flag in ("w", "a", "+")):+ raise NotImplementedError(+ "SharePoint open() write mode is not supported; "+ "use pipe_file()/write_bytes()."+ )return SharePointFile(self, path, mode, **kwargs)def cat_file(self, path: str, **kwargs: Any) -> bytes:"""Read file contents."""return self._download_file(path)- def read_bytes(self, path: str) -> bytes:- """Read file as bytes."""- return self._download_file(path)-- def write_bytes(self, path: str, data: bytes, **kwargs: Any) -> None:- """Write bytes to file."""+ def pipe_file(self, path: str, value: bytes, **kwargs: Any) -> None:+ """Write bytes to file via Graph API direct upload."""+ if not path or path.endswith("/"):+ raise ValueError(+ "pipe_file() requires a file path, not root or a directory path"+ )path = self._normalize_path(path)parts = path.split("/")name = parts[-1]parent_path = "/".join(parts[:-1]) if len(parts) > 1 else ""parent = self._get_item_by_path(parent_path)- parent.upload(name, data).execute_query()+ parent.upload(name, value).execute_query()def rm(self, path: str, recursive: bool = False, **kwargs: Any) -> None:"""Remove file or directory."""item = self._get_item_by_path(path)item.delete_object().execute_query()- def delete(self, path: str, **kwargs: Any) -> None:- """Delete file (alias for rm)."""- self.rm(path, **kwargs)-- def walk(- self,- path: str = "",- maxdepth: int | None = None,- on_error: str | Callable[[Exception], Any] = "omit",- **kwargs: Any,- ):- """Walk directory tree."""- if maxdepth is not None and maxdepth < 1:- return-- path = self._normalize_path(path)- try:- items = self.ls(path, detail=True)- except Exception as e:- if callable(on_error):- on_error(e)- elif on_error == "raise":- raise- return-- dirs = []- files = []-- for item in items:- if item["type"] == "directory":- dirs.append(item["name"].split("/")[-1])- else:- files.append(item["name"].split("/")[-1])-- yield path if path != "root" else "", dirs, files-- for d in dirs:- subpath = f"{path}/{d}" if path != "root" else d- new_depth = maxdepth - 1 if maxdepth is not None else None- yield from self.walk(subpath, maxdepth=new_depth, on_error=on_error, **kwargs)-class SharePointFS(UnstractFileSystem):"""SharePoint/OneDrive filesystem connector for Unstract.