Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

[FIX] Fix SharePoint walk() not supporting detail=True (#1843)

Chandrasekharan M9e84d45511a3
diff --git a/.claude/skills/connector-ops/SKILL.md b/.claude/skills/connector-ops/SKILL.md
index 2cbc34c25..56a315586 100644
--- a/.claude/skills/connector-ops/SKILL.md
+++ b/.claude/skills/connector-ops/SKILL.md
@@ -66,11 +66,12 @@ Ask the user for:
Use web search to discover:
1. **Official Python library** for the service
-2. **All supported authentication modes** (API key, OAuth, connection string, certificates, etc.)
-3. **Required connection parameters** for each auth mode
-4. **Official logo/icon** source
+2. **Existing fsspec provider** — check if a mature fsspec-compatible library exists (e.g., `s3fs` for S3, `adlfs` for Azure, `gcsfs` for GCS, `boxfs` for Box). **Always prefer wrapping an existing provider over building a custom one.**
+3. **All supported authentication modes** (API key, OAuth, connection string, certificates, etc.)
+4. **Required connection parameters** for each auth mode
+5. **Official logo/icon** source
-Document findings before proceeding.
+Document findings before proceeding. If no fsspec provider exists, note that a custom `AbstractFileSystem` subclass will be needed (see Step 5b).
### Step 3: Generate Connector ID
@@ -124,10 +125,33 @@ Read the template and adapt it for the specific service. Key methods to implemen
- `execute()` → Execute queries (inherited, may override)
**For Filesystem Connectors:**
-- `get_fsspec_fs()` → Return fsspec filesystem instance
+
+There are two approaches depending on whether an fsspec provider exists:
+
+**Approach 1: Wrapping an existing fsspec provider (PREFERRED)**
+When a library like `s3fs`, `adlfs`, `gcsfs`, `boxfs` exists, the connector only extends `UnstractFileSystem`:
+- `get_fsspec_fs()` → Return the provider's filesystem instance (e.g., `S3FileSystem(...)`)
- `test_credentials()` → Verify connection works
- `extract_metadata_file_hash()` → Extract file hash from metadata
- `is_dir_by_metadata()` → Check if path is directory
+- `extract_modified_date()` → Extract last modified date
+- **Do NOT implement `ls`, `walk`, `info`, etc.** — the provider handles all file operations.
+- See `minio.py` for reference.
+
+**Approach 2: Custom AbstractFileSystem (ONLY when no provider exists)**
+Create a custom `AbstractFileSystem` subclass with ONLY these core methods:
+- `ls(path, detail=True)` — List directory contents (REQUIRED)
+- `info(path)` — Get single item metadata (optimization, optional)
+- `_open(path, mode)` — Open a file for reading
+- `cat_file(path)` — Read file contents
+- `pipe_file(path, value)` — Write bytes (NOT `write_bytes` — that's an alias)
+- `rm(path)` — Delete a file
+- `mkdir(path)` — Create directory
+
+**CRITICAL: NEVER override** `listdir()`, `stat()`, `exists()`, `isdir()`, `isfile()`, `delete()`, `read_bytes()`, or `write_bytes()`. These are provided by `AbstractFileSystem` and delegate to `ls()`/`info()`/`pipe_file()`.
+
+**`walk()` exception**: Override `walk()` ONLY if the filesystem has path normalization that differs from `_strip_protocol` (e.g., SharePoint maps `""` to `"root"`), or if the service raises exceptions that aren't `FileNotFoundError`/`OSError` (the only types the base catches). If you override `walk()`, you MUST support the `detail` kwarg (return dicts when `True`, lists when `False`) and the `on_error` callback.
+- See `sharepoint.py` for reference.
**For Queue Connectors:**
- `get_engine()` → Return queue connection
@@ -467,3 +491,7 @@ Use these templates as starting points:
4. **Icon Naming**: Use PascalCase with spaces URL-encoded (e.g., `Google%20Drive.png`).
5. **Test Isolation**: Mock tests should never require external services. Use `@unittest.skipUnless` for integration tests.
+
+6. **fsspec Delegation**: NEVER override `walk()`, `listdir()`, `stat()`, `exists()`, `isdir()`, `isfile()`, `delete()`, `read_bytes()`, or `write_bytes()` in custom `AbstractFileSystem` subclasses. These are provided by fsspec's base class and delegate to `ls()`/`info()`/`pipe_file()`. Overriding them creates maintenance burden and introduces subtle bugs. See `references/connector_patterns.md` for the "Anti-Pattern: Reimplementing fsspec Base Methods" section.
+
+7. **pipe_file vs write_bytes**: When implementing file writes in a custom `AbstractFileSystem`, override `pipe_file()` (not `write_bytes()`). The base `write_bytes()` is an alias that delegates to `pipe_file()`.
diff --git a/.claude/skills/connector-ops/assets/templates/filesystem.py.template b/.claude/skills/connector-ops/assets/templates/filesystem.py.template
index 882e23f97..83d0fdfeb 100644
--- a/.claude/skills/connector-ops/assets/templates/filesystem.py.template
+++ b/.claude/skills/connector-ops/assets/templates/filesystem.py.template
@@ -1,6 +1,47 @@
"""
Filesystem Connector Template
+There are TWO approaches for filesystem connectors:
+
+## Approach 1: Wrapping an existing fsspec provider (PREFERRED)
+Use this when a mature fsspec-compatible library exists (e.g., s3fs, adlfs, gcsfs, boxfs).
+The connector class extends UnstractFileSystem and delegates all file operations to the
+provider via get_fsspec_fs(). You do NOT implement ls, walk, info, etc. — the provider
+handles all of that.
+
+## Approach 2: Custom AbstractFileSystem (RARE)
+Use this ONLY when no fsspec provider exists for the service. You must create a custom
+AbstractFileSystem subclass implementing ONLY these core methods:
+ - ls(path, detail=True) — List directory contents (REQUIRED)
+ - info(path) — Get single item metadata (optimization, optional)
+ - _open(path, mode) — Open a file for reading (REQUIRED for reads)
+ - cat_file(path) — Read file contents (REQUIRED for reads)
+ - pipe_file(path, value) — Write bytes to file (REQUIRED for writes)
+ - rm(path) — Delete a file (REQUIRED for writes)
+ - mkdir(path) — Create directory (REQUIRED for writes)
+
+NEVER override these methods — AbstractFileSystem already provides them by delegating
+to ls()/info():
+ - walk() — delegates to ls(), handles detail=True, on_error, topdown.
+ EXCEPTION: override walk() if the filesystem has custom path
+ normalization (e.g., "" → "root") or raises exceptions that aren't
+ FileNotFoundError/OSError. If overriding, MUST support detail kwarg.
+ - listdir() — alias for ls()
+ - stat() — alias for info()
+ - exists() — calls info(), catches exceptions
+ - isdir() — calls info(), checks type
+ - isfile() — calls info(), checks type
+ - delete() — alias for rm()
+ - read_bytes()— alias for cat_file()
+ - write_bytes() — alias for pipe_file()
+
+Overriding these creates maintenance burden and risks bugs (e.g., walk() not
+supporting detail=True, which breaks file discovery).
+
+Reference existing connectors:
+ - Approach 1: See minio.py (wraps s3fs.S3FileSystem)
+ - Approach 2: See sharepoint.py (custom SharePointFileSystem)
+
Replace placeholders:
- {ClassName}: PascalCase class name (e.g., MinioFS, AzureBlobFS)
- {connector_name}: lowercase connector name (e.g., minio, azure_blob)
@@ -90,8 +131,7 @@ class {ClassName}(UnstractFileSystem):
return ""
def get_fsspec_fs(self) -> AbstractFileSystem:
- """
- Return fsspec filesystem instance.
+ """Return fsspec filesystem instance.
Uses lazy initialization with thread safety for Celery compatibility.
@@ -121,8 +161,7 @@ class {ClassName}(UnstractFileSystem):
return self._fs
def test_credentials(self) -> bool:
- """
- Test filesystem credentials.
+ """Test filesystem credentials.
Returns:
True if connection successful
@@ -142,8 +181,7 @@ class {ClassName}(UnstractFileSystem):
) from e
def extract_metadata_file_hash(self, metadata: dict[str, Any]) -> str | None:
- """
- Extract unique file hash from fsspec metadata.
+ """Extract unique file hash from fsspec metadata.
Different storage systems use different keys for file hashes:
- S3/Minio: ETag
@@ -170,8 +208,7 @@ class {ClassName}(UnstractFileSystem):
return None
def is_dir_by_metadata(self, metadata: dict[str, Any]) -> bool:
- """
- Check if path is a directory from metadata.
+ """Check if path is a directory from metadata.
Args:
metadata: File metadata from fsspec
diff --git a/.claude/skills/connector-ops/references/connector_patterns.md b/.claude/skills/connector-ops/references/connector_patterns.md
index a606d67d4..2d8fa4f0b 100644
--- a/.claude/skills/connector-ops/references/connector_patterns.md
+++ b/.claude/skills/connector-ops/references/connector_patterns.md
@@ -207,6 +207,82 @@ class MyStorageFS(UnstractFileSystem):
---
+## Anti-Pattern: Reimplementing fsspec Base Methods
+
+**Problem**: When creating a custom `AbstractFileSystem` subclass (no existing fsspec
+provider), it's tempting to implement every method for "completeness". This creates
+maintenance burden and introduces bugs when the calling code evolves.
+
+**Real-world bug**: SharePoint connector overrode `walk()` but didn't support the
+`detail=True` parameter. When file discovery code switched to `walk(detail=True)`,
+SharePoint returned plain lists instead of dicts, breaking file processing silently.
+
+**Bad** — reimplementing methods the base class already provides:
+```python
+class MyFileSystem(AbstractFileSystem):
+ def ls(self, path, detail=True, **kwargs):
+ ... # Core method - MUST implement
+
+ # BAD: These are all redundant reimplementations
+ def listdir(self, path, detail=True, **kwargs):
+ return self.ls(path, detail=detail, **kwargs) # Base already does this
+
+ def stat(self, path, **kwargs):
+ return self.info(path, **kwargs) # Base already does this
+
+ def exists(self, path, **kwargs):
+ try:
+ self.info(path)
+ return True
+ except Exception:
+ return False # Base already does this
+
+ def isdir(self, path):
+ return self.info(path)["type"] == "directory" # Base already does this
+
+ def walk(self, path, maxdepth=None, **kwargs):
+ ... # Base already delegates to ls() with full detail/on_error support
+
+ def delete(self, path, **kwargs):
+ self.rm(path, **kwargs) # Base already does this
+
+ def read_bytes(self, path):
+ return self.cat_file(path) # Base already does this
+
+ def write_bytes(self, path, data, **kwargs):
+ ... # Base already delegates to pipe_file()
+```
+
+**Good** — implement only the core methods, let fsspec handle the rest:
+```python
+class MyFileSystem(AbstractFileSystem):
+ # REQUIRED: Core methods that talk to the service API
+ def ls(self, path, detail=True, **kwargs): ...
+ def info(self, path, **kwargs): ... # Optional optimization
+ def _open(self, path, mode="rb", **kwargs): ...
+ def cat_file(self, path, **kwargs): ...
+ def pipe_file(self, path, value, **kwargs): ... # NOT write_bytes
+ def rm(self, path, recursive=False, **kwargs): ...
+ def mkdir(self, path, create_parents=True, **kwargs): ...
+
+ # Everything else (walk, exists, isdir, isfile, listdir, stat,
+ # delete, read_bytes, write_bytes) is inherited from AbstractFileSystem
+```
+
+**Key insight**: `write_bytes` is an alias for `pipe_file` in fsspec. Override
+`pipe_file`, not `write_bytes`, so the delegation chain works correctly.
+
+**Exception: `walk()` may need a custom override** when the custom filesystem has
+path normalization that differs from `_strip_protocol` (e.g., SharePoint maps `""`
+to `"root"` for the Graph API), or when the service raises exceptions that aren't
+`FileNotFoundError`/`OSError` (the only types the base `walk()` catches). If you
+override `walk()`, you MUST support the `detail` kwarg (return dicts when True,
+lists when False) and the `on_error` callback.
+
+**Applies to**: Any custom `AbstractFileSystem` subclass (SharePoint, OneDrive, etc.)
+
+---
+
## Anti-Pattern: Module-Level Heavy Imports
**Bad**:
diff --git a/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py b/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py
index 9b39c67c8..7491fbf7a 100644
--- a/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py
+++ b/unstract/connectors/src/unstract/connectors/filesystems/sharepoint/sharepoint.py
@@ -11,7 +11,6 @@ Uses Microsoft Graph API via Office365-REST-Python-Client library.
import logging
import os
import threading
-from collections.abc import Callable
from datetime import UTC, datetime
from typing import Any, NoReturn
@@ -50,8 +49,9 @@ class SharePointFile(AbstractBufferedFile):
return self._file_content[start:end]
def _upload_chunk(self, final: bool = False) -> bool:
- """Upload is handled by write_bytes."""
- return True
+ raise NotImplementedError(
+ "Buffered writes not supported; use pipe_file()/write_bytes()."
+ )
class SharePointFileSystem(AbstractFileSystem):
@@ -243,10 +243,6 @@ class SharePointFileSystem(AbstractFileSystem):
logger.error(f"Error listing path {path}: {e}")
raise
- def listdir(self, path: str = "", detail: bool = True, **kwargs: Any) -> list[Any]:
- """List directory contents (alias for ls)."""
- return self.ls(path, detail=detail, **kwargs)
-
def _item_to_info(self, item: Any, parent_path: str = "") -> dict[str, Any]:
"""Convert SharePoint item to fsspec info dict."""
name = item.name
@@ -308,34 +304,6 @@ class SharePointFileSystem(AbstractFileSystem):
parent = "/".join(path.strip("/").split("/")[:-1])
return self._item_to_info(item, parent)
- def stat(self, path: str, **kwargs: Any) -> dict[str, Any]:
- """Get file/directory stats (alias for info)."""
- return self.info(path, **kwargs)
-
- def exists(self, path: str, **kwargs: Any) -> bool:
- """Check if path exists."""
- try:
- self.info(path)
- return True
- except Exception:
- return False
-
- def isdir(self, path: str) -> bool:
- """Check if path is a directory."""
- try:
- info = self.info(path)
- return info.get("type") == "directory"
- except Exception:
- return False
-
- def isfile(self, path: str) -> bool:
- """Check if path is a file."""
- try:
- info = self.info(path)
- return info.get("type") == "file"
- except Exception:
- return False
-
def mkdir(self, path: str, create_parents: bool = True, **kwargs: Any) -> None:
"""Create directory."""
path = self._normalize_path(path)
@@ -382,73 +350,37 @@ class SharePointFileSystem(AbstractFileSystem):
mode: str = "rb",
**kwargs: Any,
) -> SharePointFile:
- """Open a file."""
+ """Open a file for reading. Write modes are not supported."""
+ if any(flag in mode for flag in ("w", "a", "+")):
+ raise NotImplementedError(
+ "SharePoint open() write mode is not supported; "
+ "use pipe_file()/write_bytes()."
+ )
return SharePointFile(self, path, mode, **kwargs)
def cat_file(self, path: str, **kwargs: Any) -> bytes:
"""Read file contents."""
return self._download_file(path)
- def read_bytes(self, path: str) -> bytes:
- """Read file as bytes."""
- return self._download_file(path)
-
- def write_bytes(self, path: str, data: bytes, **kwargs: Any) -> None:
- """Write bytes to file."""
+ def pipe_file(self, path: str, value: bytes, **kwargs: Any) -> None:
+ """Write bytes to file via Graph API direct upload."""
+ if not path or path.endswith("/"):
+ raise ValueError(
+ "pipe_file() requires a file path, not root or a directory path"
+ )
path = self._normalize_path(path)
parts = path.split("/")
name = parts[-1]
parent_path = "/".join(parts[:-1]) if len(parts) > 1 else ""
parent = self._get_item_by_path(parent_path)
- parent.upload(name, data).execute_query()
+ parent.upload(name, value).execute_query()
def rm(self, path: str, recursive: bool = False, **kwargs: Any) -> None:
"""Remove file or directory."""
item = self._get_item_by_path(path)
item.delete_object().execute_query()
- def delete(self, path: str, **kwargs: Any) -> None:
- """Delete file (alias for rm)."""
- self.rm(path, **kwargs)
-
- def walk(
- self,
- path: str = "",
- maxdepth: int | None = None,
- on_error: str | Callable[[Exception], Any] = "omit",
- **kwargs: Any,
- ):
- """Walk directory tree."""
- if maxdepth is not None and maxdepth < 1:
- return
-
- path = self._normalize_path(path)
- try:
- items = self.ls(path, detail=True)
- except Exception as e:
- if callable(on_error):
- on_error(e)
- elif on_error == "raise":
- raise
- return
-
- dirs = []
- files = []
-
- for item in items:
- if item["type"] == "directory":
- dirs.append(item["name"].split("/")[-1])
- else:
- files.append(item["name"].split("/")[-1])
-
- yield path if path != "root" else "", dirs, files
-
- for d in dirs:
- subpath = f"{path}/{d}" if path != "root" else d
- new_depth = maxdepth - 1 if maxdepth is not None else None
- yield from self.walk(subpath, maxdepth=new_depth, on_error=on_error, **kwargs)
-
class SharePointFS(UnstractFileSystem):
"""SharePoint/OneDrive filesystem connector for Unstract.