Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

21.9 KB
Raw Blame

title: "Ingest Module" description: "Universal data ingestion from files, Parquet, XML, web, public APIs, feeds, streams, repositories, email, and databases." icon: "database"

semantica.ingest is the universal entry point for loading data into Semantica:

  • 15+ ingestion adapters: files, web, SQL, Databricks, Snowflake, Kafka, MCP, Git repos, email
  • PyArrow Parquet with column selection and partitioned dataset support
  • XXE-safe lxml XML with optional XSD schema validation
  • ingest() unified dispatcher: auto-detects source type from path or URL
  • Each ingestor returns its own typed object (FileObject, WebContent, TableData, etc.)

Exported Classes

Class Role
FileIngestor PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR: type auto-detected from extension
CloudStorageIngestor Unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage
WebIngestor Web scraping and crawling with ingest_url, crawl_sitemap, crawl_domain
RESTIngestor Generic REST API ingestion with headers, params, retries, and pagination
PublicAPIIngestor No-auth public API ingestion with pre-configured examples and rate limiting
FeedIngestor RSS/Atom feed ingestion with live monitoring via FeedMonitor
StreamIngestor Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar
RepoIngestor Git repositories: source files, commit history, and metadata
DBIngestor SQL databases via SQLAlchemy: tables, views, and custom queries
SnowflakeIngestor Snowflake data warehouse queries and table exports
DatabricksIngestor Databricks Unity Catalog metadata, Delta table queries, and lineage
SAPIngestor SAP OData services (S/4HANA Cloud, SuccessFactors, NetWeaver Gateway): entity-set discovery and ingestion with v2/v4 pagination
ParquetIngestor Apache Parquet files and partitioned datasets with column selection
ArrowIngestor Apache Arrow IPC and Feather file processing
XMLIngestor XXE-safe XML parsing with optional XSD schema validation
EmailIngestor IMAP/POP3 email ingestion with attachment extraction
OntologyIngestor OWL/RDF/Turtle ontology file ingestion
MCPIngestor Model Context Protocol (MCP) resource ingestion
ingest() Unified dispatcher: detects source type automatically from path or URL

Getting Started

Use FileIngestor for local files: it auto-detects format from the file extension and handles archives:

from semantica.ingest import FileIngestor

ingestor = FileIngestor()

# Single file -> FileObject
file_obj = ingestor.ingest_file("data/report.pdf")
print(file_obj.name)       # "report.pdf"
print(file_obj.file_type)  # "pdf"
print(file_obj.text)       # decoded text content (property on FileObject)
print(file_obj.size)       # bytes

# Directory scan -> List[FileObject]
files = ingestor.ingest_directory("data/", recursive=True)
for f in files:
    print(f.name, f.file_type, f.size)
**`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and reads content into `.content` bytes or the `.text` property. Use `read_content=False` when you only need file metadata.

For web, database, or stream sources, each ingestor exposes its own typed method:

# Web
from semantica.ingest import WebIngestor
wc = WebIngestor(delay=1.0, respect_robots=True).ingest_url("https://example.com")
print(wc.title, wc.text)

# Database: constructor takes no required args; pass connection_string to methods
from semantica.ingest import DBIngestor
db = DBIngestor()
result = db.ingest_database("postgresql://user:pass@localhost/db")
# result["tables"]["documents"]["rows"] contains the rows

# Unified dispatcher: auto-detects source type
from semantica.ingest import ingest
result = ingest("data/report.pdf")          # -> {"files": [FileObject]}
result = ingest("https://example.com")      # -> {"content": WebContent}
result = ingest("data/events.parquet")      # -> {"data": ParquetData}
result = ingest("ontology.ttl")             # -> {"ontology": OntologyData}

Quick Start

```python from semantica.ingest import FileIngestor
ingestor = FileIngestor()

# Single file: type auto-detected from extension
file_obj = ingestor.ingest_file("data/report.pdf")

# Recursive directory scan
files = ingestor.ingest_directory("data/", recursive=True)

# ingest() also works: routes to file or directory automatically
from semantica.ingest import ingest
result = ingest("data/report.pdf")   # {"files": [FileObject]}
```
```python from semantica.ingest import DBIngestor
ingestor = DBIngestor()

# Ingest all tables
result = ingestor.ingest_database(
    "postgresql://user:pass@localhost/db",
    include_tables=["documents"],
)
# result["tables"]["documents"]["rows"] contains the row dicts

# Run a custom query
rows = ingestor.execute_query(
    "postgresql://user:pass@localhost/db",
    "SELECT id, content, created_at FROM documents WHERE status = :s",
    s="active",
)
```
```python from semantica.ingest import FileIngestor from semantica.pipeline import PipelineBuilder, ExecutionEngine from semantica.parse import DocumentParser from semantica.semantic_extract import NERExtractor
ingestor  = FileIngestor()
parser    = DocumentParser()
extractor = NERExtractor(method="ml")

builder = PipelineBuilder()
builder.add_step("ingest",  "file_ingest",    handler=ingestor.ingest_file)
builder.add_step("parse",   "document_parse", handler=parser.parse)
builder.add_step("extract", "ner_extract",    handler=extractor.extract)
builder.connect_steps("ingest", "parse")
builder.connect_steps("parse",  "extract")

pipeline = builder.build("my_pipeline")
result   = ExecutionEngine().execute_pipeline(pipeline, data="data/report.pdf")
```

Ingestors

### FileIngestor
```python
from semantica.ingest import FileIngestor

ingestor = FileIngestor()

# Single file
file_obj = ingestor.ingest_file("data/report.pdf")

# Directory: returns List[FileObject]
files = ingestor.ingest_directory("data/", recursive=True)

# ingest() dispatches to ingest_file or ingest_directory automatically
files = ingestor.ingest("data/")
```

Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives.

<Note>
  Glob patterns (e.g. `"data/**/*.docx"`) are **not** supported. `ingest()` accepts a file path or a directory path only. To filter by extension inside a directory, use `ingest_directory()` with the `pattern=` filter option.
</Note>

### ParquetIngestor

PyArrow-based ingestion for Apache Parquet files, including Hive-style partitioned datasets:

```python
from semantica.ingest import ParquetIngestor

ingestor = ParquetIngestor()

# Single Parquet file -> ParquetData
data = ingestor.ingest_file("data/events.parquet")

# Partitioned directory (year=2024/month=01/...)
data = ingestor.ingest_directory("data/partitioned/")

# Load only specific columns: pass as kwarg
from semantica.ingest import ingest_parquet
data = ingest_parquet("data/events.parquet", columns=["id", "text", "timestamp"])

# Extract schema without loading data
schema = ingest_parquet("data/events.parquet", method="schema")
```

Requires `pyarrow`: `pip install pyarrow`.

<Tip>
  **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns.
</Tip>

### XMLIngestor

XXE-safe lxml-based ingestion with optional schema validation:

```python
from semantica.ingest import XMLIngestor

# Basic ingestion
ingestor = XMLIngestor()
data = ingestor.ingest_file("data/records.xml")

# With XSD validation: pass schema_path as kwarg
from semantica.ingest import ingest_xml
data = ingest_xml("data/records.xml", schema_path="schema.xsd")

# Validation report only
report = ingest_xml("data/feed.xml", method="validate", schema_path="schema.xsd")

# Directory scan
results = ingestor.ingest_directory("data/records/")
```

<Note>
  `XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks.
</Note>

<Warning>
  **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
</Warning>
### WebIngestor
```python
from semantica.ingest import WebIngestor

ingestor = WebIngestor(
    delay=1.0,             # seconds between requests
    respect_robots=True,   # honor robots.txt
    timeout=30,
)

# Single URL -> WebContent
content = ingestor.ingest_url("https://example.com/about")
print(content.title)
print(content.text)
print(content.links)

# Sitemap crawl -> List[WebContent]
pages = ingestor.crawl_sitemap("https://example.com/sitemap.xml")

# Domain crawl -> List[WebContent]
pages = ingestor.crawl_domain("https://example.com", max_pages=50)
```

Requires `beautifulsoup4`: `pip install beautifulsoup4`.

<Tip>
  **Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting you risk getting blocked by the target server or violating its terms of service.
</Tip>

### PublicAPIIngestor

Use this for public REST-style APIs that do not require keys or tokens:

```python
from semantica.ingest import PublicAPIIngestor, PublicAPIExamples, ingest_public_api

ingestor = PublicAPIIngestor(rate_limit_delay=1.0)

# Ingest any public endpoint
data = ingestor.ingest_public_api("https://jsonplaceholder.typicode.com/posts")

# Use a pre-configured example by name
data = ingestor.ingest_example("rest_countries_all")

# Check if endpoint is accessible without auth
detection = ingestor.detect_public_api("https://jsonplaceholder.typicode.com/posts")

# List available pre-configured examples
examples = PublicAPIExamples.list_examples()

# Convenience function
data = ingest_public_api("https://jsonplaceholder.typicode.com/posts")
```

Public API ingestion rejects common auth headers and query parameters by
default. Use `RESTIngestor` for authenticated APIs.

### FeedIngestor (RSS/Atom)

```python
from semantica.ingest import FeedIngestor

ingestor = FeedIngestor()

# Ingest a feed -> FeedData
feed = ingestor.ingest_feed("https://feeds.example.com/rss")

# Discover feeds from a website
from semantica.ingest import ingest_feed
feeds = ingest_feed("https://example.com", method="discover")
```

Requires `beautifulsoup4`: `pip install beautifulsoup4`.

### RepoIngestor

Ingest Git repositories: source code, commit history, and dependency graphs:

```python
from semantica.ingest import RepoIngestor

ingestor = RepoIngestor(
    branch="main",
    file_types=[".py", ".md", ".yaml"],
    include_commits=True,
    commit_range="HEAD~100..HEAD",
)

result = ingestor.ingest_repository("https://github.com/org/repo")
result = ingestor.ingest_repository("/path/to/local/repo")
```

Requires `GitPython`: `pip install gitpython`.

### EmailIngestor

Ingest emails via IMAP or POP3 with attachment extraction and thread analysis:

```python
from semantica.ingest import EmailIngestor
import os

ingestor = EmailIngestor(
    protocol="imap",
    host="imap.gmail.com",
    port=993,
    use_ssl=True,
    username=os.getenv("EMAIL_USER"),
    password=os.getenv("EMAIL_PASS"),
    folder="INBOX",
    attachment_types=[".pdf", ".docx", ".txt"],
    include_thread_analysis=True,
    max_emails=500,
)
emails = ingestor.ingest()
```

Requires `beautifulsoup4`: `pip install beautifulsoup4`.
### CloudStorageIngestor
`CloudStorageIngestor` is a unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage:

```python
from semantica.ingest import CloudStorageIngestor
import os

# AWS S3: list and download objects
ingestor = CloudStorageIngestor(
    provider="s3",
    access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    region="us-east-1",
)
objects = ingestor.list_objects("my-documents-bucket", prefix="reports/2024/")
content = ingestor.download_object("my-documents-bucket", "reports/2024/report.pdf")

# FileIngestor.ingest_cloud() wraps CloudStorageIngestor
from semantica.ingest import FileIngestor
files = FileIngestor().ingest_cloud(
    provider="s3",
    bucket="my-documents-bucket",
    prefix="reports/2024/",
    access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    region="us-east-1",
)
```
### DBIngestor (SQL)
`DBIngestor` takes no required constructor args. Pass the connection string to each method:

```python
from semantica.ingest import DBIngestor

ingestor = DBIngestor()

# Ingest entire database (all tables, or filtered)
result = ingestor.ingest_database(
    "postgresql://user:pass@localhost/db",
    include_tables=["documents"],
)
# result["schema"], result["tables"], result["total_tables"]

# Run a custom query -> List[Dict]
rows = ingestor.execute_query(
    "postgresql://user:pass@localhost/db",
    "SELECT id, content FROM documents WHERE status = :s",
    s="active",
)

# Export a single table -> TableData
table = ingestor.export_table(
    "postgresql://user:pass@localhost/db",
    table_name="documents",
    limit=1000,
)
```

Requires `sqlalchemy`: `pip install sqlalchemy` plus your database driver.

<Warning>
  **`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself.
</Warning>

### SnowflakeIngestor

```python
from semantica.ingest import SnowflakeIngestor
import os

ingestor = SnowflakeIngestor(
    account=os.getenv("SNOWFLAKE_ACCOUNT"),
    user=os.getenv("SNOWFLAKE_USER"),
    password=os.getenv("SNOWFLAKE_PASSWORD"),
    warehouse="COMPUTE_WH",
    database="ANALYTICS",
    schema="PUBLIC",
)
result = ingestor.ingest_query("SELECT * FROM documents")
result = ingestor.ingest_table("documents")
```

### DatabricksIngestor

```python
from semantica.ingest import DatabricksIngestor
import os

ingestor = DatabricksIngestor(
    host=os.getenv("DATABRICKS_HOST"),
    token=os.getenv("DATABRICKS_TOKEN"),
    http_path=os.getenv("DATABRICKS_HTTP_PATH"),
    catalog="main",
    schema="default",
)
result = ingestor.ingest_query("SELECT * FROM documents")
result = ingestor.ingest_table("documents")
lineage = ingestor.get_table_lineage("documents")
```
### StreamIngestor
Real-time ingestion from message brokers: each method returns a typed processor:

```python
from semantica.ingest import StreamIngestor

ingestor = StreamIngestor()

# Kafka -> KafkaProcessor
processor = ingestor.ingest_kafka(
    topic="documents",
    bootstrap_servers=["localhost:9092"],
)
processor.set_message_handler(lambda msg: print(msg))
processor.start_consuming()

# RabbitMQ -> RabbitMQProcessor
processor = ingestor.ingest_rabbitmq(
    queue="document_queue",
    connection_url="amqp://guest:guest@localhost/",
)

# AWS Kinesis -> KinesisProcessor
processor = ingestor.ingest_kinesis(
    stream_name="documents-stream",
    region="us-east-1",
)

# Apache Pulsar -> PulsarProcessor
processor = ingestor.ingest_pulsar(
    topic="persistent://public/default/documents",
    service_url="pulsar://localhost:6650",
)

# Start all processors at once
ingestor.start_streaming()
# Stop all processors
ingestor.stop_streaming()

# Monitor stream health
health = ingestor.monitor.check_health()
```

Stream processors require the appropriate client library (kafka-python, pika, boto3, pulsar-client).

<Warning>
  **`StreamIngestor` methods require the target broker's client library to be installed.** `ingest_kafka` needs `kafka-python`, `ingest_rabbitmq` needs `pika`, `ingest_kinesis` needs `boto3`, and `ingest_pulsar` needs `pulsar-client`. Missing dependencies raise `ImportError` at call time, not at import time.
</Warning>

ingest() Unified Dispatcher

ingest() auto-detects source type from the path or URL and routes to the appropriate ingestor. It returns a Dict[str, Any] where the key depends on source type:

from semantica.ingest import ingest

# File
result = ingest("report.pdf")               # {"files": [FileObject]}
result = ingest("data/", source_type="file") # {"files": [FileObject, ...]}

# Web
result = ingest("https://example.com")       # {"content": WebContent}

# Feed (auto-detected from URL pattern)
result = ingest("https://example.com/feed.xml") # {"feeds": FeedData}

# Parquet (auto-detected from .parquet extension)
result = ingest("events.parquet")            # {"data": ParquetData}

# XML (auto-detected from .xml extension)
result = ingest("records.xml")               # {"xml": XMLIngestionData}

# Ontology (auto-detected from .ttl/.owl/.rdf)
result = ingest("ontology.ttl")              # {"ontology": OntologyData}

# Database (auto-detected from connection string prefix)
result = ingest("postgresql://user:pass@localhost/db") # {"data": ...}

# Public API
result = ingest(
    "https://jsonplaceholder.typicode.com/posts",
    source_type="public_api",
)                                            # {"data": APIData}

ingest() Parameters

Parameter Type Default Description
sources str, Path, or List required File path, URL, directory, or connection string
source_type str None (auto-detected) "file", "web", "public_api", "feed", "stream", "repo", "email", "db", "parquet", "xml", "ontology", "mcp"
method str None Optional method override passed to the underlying ingestor
**kwargs Extra options forwarded to the underlying ingestor method

FileObject Fields

FileIngestor returns FileObject instances:

from dataclasses import dataclass
from datetime import datetime
from typing import Any, Dict, Optional

@dataclass
class FileObject:
    path:        str                    # absolute file path
    name:        str                    # filename (e.g. "report.pdf")
    size:        int                    # size in bytes
    file_type:   str                    # detected type without dot (e.g. "pdf", "docx")
    mime_type:   Optional[str]          # MIME type if detectable
    content:     Optional[bytes]        # raw bytes (None if read_content=False)
    metadata:    Dict[str, Any]         # extension, parent dir, is_supported, etc.
    ingested_at: datetime               # ingestion timestamp

    @property
    def text(self) -> str:
        """Decoded text from content bytes (UTF-8 with latin-1 fallback)."""
        ...

To get text from an ingested file, use the .text property:

file_obj = FileIngestor().ingest_file("report.pdf")
text = file_obj.text       # decoded string
raw  = file_obj.content    # raw bytes

Skip reading content (useful for directory scanning without loading files):

files = FileIngestor().ingest_directory("data/", recursive=True, read_content=False)

OntologyIngestor

Ingest existing OWL or RDF ontology files as structured knowledge sources:

from semantica.ingest import OntologyIngestor

ingestor = OntologyIngestor()

data = ingestor.ingest_ontology("domain_ontology.owl", format="turtle")

# Or using the convenience function
from semantica.ingest import ingest_ontology
data = ingest_ontology("domain_ontology.ttl")

Custom Ingestors

Register a custom ingestor function to participate in the full registry:

from semantica.ingest.registry import method_registry
from semantica.ingest import FileObject

def my_ingestor(source, **kwargs):
    # Return whatever your format produces
    return FileObject(
        path=source,
        name=source,
        size=0,
        file_type="custom",
        content=b"...",
        metadata={},
    )

method_registry.register("file", "my_format", my_ingestor)

# Now callable via the convenience function
from semantica.ingest import ingest_file
result = ingest_file("source_path", method="my_format")
  • Parse — Parse raw sources into structured text and tables.
  • Pipeline — Orchestrate ingest as the first pipeline step.
  • Snowflake Integration — Snowflake-specific setup and authentication guide.
  • Databricks Integration — Databricks Unity Catalog setup, authentication, and lineage guide.
  • Provenance — Track lineage from ingest through to inference.