Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Fix language spec examples and operator details (#471)
Louis Choquel110f5850e769
diff --git a/MANIFESTO.md b/MANIFESTO.mddeleted file mode 100644index a6abe44db..000000000--- a/MANIFESTO.md+++ /dev/null@@ -1,61 +0,0 @@-### This is our manifesto, this is why we built [Pipelex](https://pipelex.com).--First published on our [blog](https://www.pipelex.com/post/repeatable-ai-workflows-knowledge-pipelines) on June 3, 2025.------# The Knowledge Pipeline Manifesto--## Agents are brilliant but they're hopeless at repeatability--AI agents are getting impressive at novel tasks: research, planning, coding from scratch. But here's what we're missing: lots of knowledge work isn't novel. It's the same patterns, repeated thousands of times.--Ask an agent to review 10,000 expense reports. Watch it improvise a new approach for each one. Different reasoning paths, different output formats, different edge case handling. Run it twice on the same report? Different results. The intelligence is there, but the consistency isn't. And **someone’s paying for those improv tokens.**--Agents are explorers. But if you know the path, you don't need an expedition. You need a bullet train.--## Enter the knowledge pipeline--The solution should be obvious: delegate the repetitive tasks to deterministic AI workflows. But here's the deeper problem: AI workflows today are handcrafted, not engineered. We prompt, tweak, test, and pray.--Without proper tools for capturing repeatable workflows, every company recreates the same solutions from scratch. The expense categorization that took your team weeks to perfect? The next company starts at zero. We're in the pre-SQL era of AI: brilliant capabilities trapped in proprietary silos, no way to reuse what actually works.--We need a way to capture proven AI workflows and make them as reliable and reusable as any other software component. We call this a **knowledge pipeline.**--A knowledge pipeline is built from modular components called pipes. Each pipe is a knowledge transformer with a simple contract: **knowledge in, knowledge out**. Unlike data pipelines that move bytes or ML pipelines that train models, knowledge pipes transform *meaning*.--This isn't about dumbing down AI. Each pipe leverages full AI intelligence to handle variation in content while guaranteeing its output structure. You get **deterministic structure with adaptive intelligence:** consistency without rigidity.--## The architecture of understanding--### **Knowledge pipes compose like LEGO blocks for repeatable AI workflows:**--- Connect them sequentially for step-by-step transformations-- Run them in parallel to process multiple perspectives-- Feed multiple outputs into a single pipe for synthesis-- Call sub-pipes conditionally based on categorization--Why does this matter? Because real knowledge work isn't linear.--A pipe analyzing contracts might need context from step 2, results from steps 5-7, and external compliance data. The architecture mirrors how knowledge actually flows: not as a conveyor belt, but as a network of understanding.--## The method becomes the artifact--When every pipe guarantees its output structure, you can compose them fearlessly. When each pipe uses AI intelligently, the pipeline adapts without breaking. **When you capture your method as actual code, you can version it, test it, and scale it like software.**--With knowledge pipelines, your process becomes something you can hold, examine, and refine. **Swap out language models to find the sweet spot between cost and quality.** Adjust prompts until they sing. Reconfigure the flow when you discover a shortcut. Each iteration teaches you something new because you're working with stable, measurable outcomes.--Remember when ChatGPT launched and suddenly everyone was sharing prompts? That same collaborative instinct is waiting to be unleashed on knowledge pipelines. A pipeline for extracting customer insights gets forked for user research. A financial audit workflow becomes the skeleton for compliance reviews. The building blocks translate across industries.--These pipeline artifacts are meant to be shared, forked, and composed. Period.--## The path forward--We've seen this movie before. Docker transformed deployment by giving us a simple way to package and share what used to live in scattered shell scripts and tribal knowledge. Suddenly, complex configurations became artifacts you could version, test, and run anywhere. Better yet, Docker made it possible to build on each other's work rather than starting from scratch each time.--Today's AI workflows are ready for the same breakthrough.--This is why we built [**Pipelex**](https://github.com/Pipelex/pipelex) as an open-source standard for repeatable AI workflows. Not just another tool, but a foundation for the community to build on.--The manifesto is simple: **When you discover a method that works, capture it as a knowledge pipeline. When you build a pipeline that roars, share it with the world!**--Join us in building the knowledge infrastructure of tomorrow. One pipe at a time.diff --git a/Makefile b/Makefileindex a8fc7c13f..f47291b81 100644--- a/Makefile+++ b/Makefile@@ -538,7 +538,7 @@ docs-check: envdocs-deploy: env$(call PRINT_TITLE,"Deploying documentation with mkdocs")- $(VENV_MKDOCS) gh-deploy --force --clean+ $(VENV_MKDOCS) gh-deploy --force --clean -s############################################################################################# SHORTHANDSdiff --git a/docs/home/3-understand-pipelex/language-spec-v0-1-0.md b/docs/home/3-understand-pipelex/language-spec-v0-1-0.mdnew file mode 100644index 000000000..eb841795d--- /dev/null+++ b/docs/home/3-understand-pipelex/language-spec-v0-1-0.md@@ -0,0 +1,184 @@+# Pipelex (PLX) – Declarative AI Workflow Spec (v0.1.0)++**Build deterministic, repeatable AI workflows using declarative TOML syntax.**++The Pipelex Language (PLX) uses a TOML-based syntax to define deterministic, repeatable AI workflows. This specification documents version 0.1.0 of the language and establishes the canonical way to declare domains, concepts, and pipes inside `.plx` bundles.++---++## Core Idea++Pipelex is a workflow declaration language that gets interpreted at runtime, we already have a Python runtime (see [github.com/pipelex/pipelex](https://github.com/pipelex/pipelex)).++Pipelex lets you declare **what** your AI workflow should accomplish and **how** to execute it step by step. Each `.plx` file represents a bundle where you define:++- **Concepts** (PascalCase): the structured or unstructured data flowing through your system+- **Pipes** (snake_case): operations or orchestrators that define your workflow+- **Domain** (named in snake_case): the topic or field of work this bundle is about++Write once in `.plx` files. Run anywhere. Get the same results every time.++---++## Semantics++Pipelex workflows are **declarative and deterministic**:++- Pipes are evaluated based on their dependencies, not declaration order+- Controllers explicitly define execution flow (sequential, parallel, or conditional)++All concepts are strongly typed. All pipes declare their inputs and outputs. The runtime validates that data flowing between pipes matches the declared types before execution. Concepts are also conceptually defined, adding meaning on top of the technical typing.++---++## Guarantees & Limits++**Guarantees:**++- Deterministic workflow execution and outputs+- Strong typing with validation before runtime++**Not supported in v0.1.0:**++- Pipe namespaces per domain++**More:**++- GitHub: [github.com/pipelex/pipelex](https://github.com/pipelex/pipelex)++---++## Complete Example: CV Job Matching Workflow++This workflow analyses candidate CVs against job offer requirements to determine match quality.++```toml+domain = "cv_job_matching"+description = "Analyzing candidate CVs against job offer requirements to determine match quality."+main_pipe = "analyze_cv_job_matches"++[concept.JobRequirements]+description = "Structured representation of a job position's requirements and details."++[concept.JobRequirements.structure]+role_title = { type = "text", description = "The job title or position name", required = true }+key_responsibilities = { type = "text", description = "Main duties and responsibilities of the role", required = true }+required_qualifications = { type = "text", description = "Mandatory qualifications, skills, or experience needed", required = true }+preferred_qualifications = { type = "text", description = "Nice-to-have qualifications, skills, or experience" }+experience_level = { type = "text", description = "Required years or level of experience" }+technical_skills = { type = "text", description = "Specific technical skills or tools required" }+soft_skills = { type = "text", description = "Interpersonal or behavioral skills desired" }++[concept.CvMatchAnalysis]+description = "Assessment of how well a candidate's CV matches job requirements."++[concept.CvMatchAnalysis.structure]+overall_match_score = { type = "number", description = "Numerical score representing overall fit (0-100)", required = true }+strengths = { type = "text", description = "Areas where the candidate excels or meets requirements well", required = true }+weaknesses = { type = "text", description = "Areas where the candidate falls short or lacks requirements", required = true }+recommendation = { type = "text", description = "Final recommendation on whether to proceed with the candidate", required = true }+relevant_experience = { type = "text", description = "Summary of candidate's relevant work experience" }+skills_match = { type = "text", description = "Analysis of how candidate's skills align with job requirements" }++[pipe.analyze_cv_job_matches]+type = "PipeSequence"+description = """+Main pipeline that extracts job requirements from the job offer PDF and analyzes all candidate CVs against those requirements to determine match quality.+"""+inputs = { job_offer_pdf = "PDF", cv_pdfs = "PDF[]" }+output = "CvMatchAnalysis[]"+steps = [+ { pipe = "extract_job_offer", result = "job_offer_pages" },+ { pipe = "structure_job_requirements", result = "job_requirements" },+ { pipe = "process_all_cvs", result = "cv_match_analyses" },+]++[pipe.extract_job_offer]+type = "PipeExtract"+description = "Extracts text content from the job offer PDF document into pages."+inputs = { job_offer_pdf = "PDF" }+output = "Page[]"+model = "extract_text_from_pdf"++[pipe.structure_job_requirements]+type = "PipeLLM"+description = """+Consolidates all pages from the job offer into a single structured job description with key requirements, qualifications, and responsibilities.+"""+inputs = { job_offer_pages = "Page[]" }+output = "JobRequirements"+model = "llm_to_retrieve"+system_prompt = """+You are an expert HR analyst specializing in extracting and structuring job requirements from job postings. Your task is to produce a structured JobRequirements object based on the provided job offer document.+"""+prompt = """+Analyze the following job offer pages and extract the key information into a structured format.++@job_offer_pages++Extract and organize the job requirements, including the role title, key responsibilities, required and preferred qualifications, experience level, technical skills, and soft skills.+"""++[pipe.process_all_cvs]+type = "PipeBatch"+description = "Processes each CV PDF concurrently to extract and analyze against job requirements."+inputs = { cv_pdfs = "PDF[]", job_requirements = "JobRequirements" }+output = "CvMatchAnalysis[]"+branch_pipe_code = "process_single_cv"+input_list_name = "cv_pdfs"+input_item_name = "cv_pdf"++[pipe.process_single_cv]+type = "PipeSequence"+description = "Processes a single CV through extraction and analysis against job requirements."+inputs = { cv_pdf = "PDF", job_requirements = "JobRequirements" }+output = "CvMatchAnalysis"+steps = [+ { pipe = "extract_cv_content", result = "cv_pages" },+ { pipe = "analyze_cv_match", result = "cv_match_analysis" },+]++[pipe.extract_cv_content]+type = "PipeExtract"+description = "Extracts text content from a single CV PDF into pages."+inputs = { cv_pdf = "PDF" }+output = "Page[]"+model = "extract_text_from_pdf"++[pipe.analyze_cv_match]+type = "PipeLLM"+description = """+Analyzes the CV content against the job requirements and produces a match assessment with scores, strengths, weaknesses, and overall recommendation.+"""+inputs = { cv_pages = "Page[]", job_requirements = "JobRequirements" }+output = "CvMatchAnalysis"+model = "llm_to_answer_questions"+system_prompt = """+You are an expert HR analyst and recruiter. Your task is to analyze candidate CVs against job requirements and produce structured assessments. Be thorough, objective, and provide actionable insights.+"""+prompt = """+Analyze the following CV against the job requirements and provide a comprehensive match assessment.++@cv_pages++Job Requirements:+- Role: $job_requirements.role_title+- Key Responsibilities: $job_requirements.key_responsibilities+- Required Qualifications: $job_requirements.required_qualifications+- Preferred Qualifications: $job_requirements.preferred_qualifications+- Experience Level: $job_requirements.experience_level+- Technical Skills: $job_requirements.technical_skills+- Soft Skills: $job_requirements.soft_skills++Evaluate how well this candidate matches the job requirements.+"""+```++**What happens:**++- Extracts and structures job requirements from a job offer PDF using document extraction + LLM+- Processes all candidate CVs in parallel (batch processing)+- Each CV is extracted and analyzed against the structured job requirements using an LLM+- Produces a scored match analysis for each candidate with strengths, weaknesses, and hiring recommendations+- Demonstrates sequential orchestration, parallel processing, nested workflows, and strong typing+diff --git a/docs/home/3-understand-pipelex/manifesto.md b/docs/home/3-understand-pipelex/manifesto.mddeleted file mode 100644index a5cc3de90..000000000--- a/docs/home/3-understand-pipelex/manifesto.md+++ /dev/null@@ -1,5 +0,0 @@-----title: The Knowledge Pipeline Manifesto--------8<-- "MANIFESTO.md"diff --git a/docs/home/3-understand-pipelex/viewpoint.md b/docs/home/3-understand-pipelex/viewpoint.mdnew file mode 100644index 000000000..90d4117a2--- /dev/null+++ b/docs/home/3-understand-pipelex/viewpoint.md@@ -0,0 +1,147 @@+### This is our viewpoint, this is why we built [Pipelex](https://pipelex.com).++Web version: https://knowhowgraph.com/++---+# Viewpoint: The Know-How Graph++Declarative, Repeatable AI Workflows as Shared Infrastructure++**TL;DR**++Agents are great at solving new problems, terrible at doing the same thing twice.++We argue that repeatable AI workflows should complement agents: written in a declarative language that both humans and agents can understand, reuse, and compose. These workflows become tools that agents can build, invoke, and share to turn repeatable cognitive work into reliable infrastructure.++At scale, this forms a **Know-How Graph:** a network of reusable methods that become shared infrastructure.++## **Agents Are Brilliant but Hopeless at Repeatability**++When an AI agent encounters the same task for the nth time, it behaves as if it were the first: different reasoning paths, different output structures, different edge-case handling. **This is fine for novel problems, but wasteful for proven processes.**++### **Agents remember knowledge, but not know-how**++This is **the repeatability paradox**. Agents excel at understanding requirements and designing solutions. They can reason, analyze a task, break it down into logical steps, model data structures. They can remember facts and past conversations. But they can't remember the method they used to solve that same task last time. **They end up reinventing the approach every time, and someone's paying for those improv tokens.**++### We Need a Standard for Reusable Methods++The solution is to capture these methods as AI workflows so agents can reuse them.++By "AI workflows" we mean the actual intellectual work that wasn't automatable before LLMs: extracting structured data from unstructured documents, applying complex analyses and business rules, generating reports with reasoning. **This isn’t about API plumbing or app connectors, it’s about the actual intellectual work.**++Yet look at what's happening today: teams everywhere are hand-crafting the same workflows from scratch. To extract data points from contracts and RFPs, to process expense reports, to classify documents, to screen resumes: identical problems solved in isolation, burning engineering hours.++## AI workflows must be formalized++OpenAPI and MCP enable interoperability for software and agents. The remaining problem is formalizing the **methods that assemble the cognitive steps themselves:** extraction, analysis, synthesis, creativity, and decision-making, the part where understanding matters. These formalized methods must be:++- **Consistent:** same input, same output, every time.+- **Efficient:** use the right AI model for each step, large or small.+- **Transparent:** no black boxes. Domain experts can audit the logic, spot issues, suggest improvements.++The workflow becomes a shared artifact that humans and AI collaborate on, optimize together, and trust to run at scale.++### Current solutions are inadequate++Engineers building AI workflows today are stuck with bad options.++Code frameworks like LangChain require **maintaining custom software for every workflow,** with business logic buried in implementation details and technical debt accumulating with each new use case.++Visual builders like Zapier, Make, or n8n excel at what they're designed for: connecting APIs and automating data flow between services. **But automation platforms are not cognitive workflow systems.** AI was bolted on as a feature after the fact. They weren't built for intellectual work. When you need actual understanding and multi-step reasoning, these tools quickly become unwieldy.++None of these solutions speak the language of the domain expert. None of them were built for agents to understand, modify, or generate workflows from requirements. They express technical plumbing, not business logic.++At the opposite, agent SDKs and multi-agent frameworks give you flexibility but sacrifice the repeatability you need for production. **You want agents for exploration and problem-solving, but when you've found a solution that works, you need to lock it down.**++> We need a universal workflow language that expresses business logic, not technical plumbing.+This workflow language must run across platforms, models, and agent frameworks, where the method outlives any vendor or model version.+>++## We Need a Declarative Language++AI workflows should be first-class citizens of our technical infrastructure: not buried in code or trapped in platforms, but expressed in a language built for the job. The method should be an artifact you can version, diff, test, and optimize.++**We need a declarative language that states what you want, not how to compute it.** As SQL separated intent from implementation for data, we need the same for AI workflows — so we can build a Know-How Graph: a reusable graph of methods that agents and humans both understand.++### The language shouldn’t need documentation: it is the documentation++Traditional programs are instructions a machine blindly executes. The machine doesn’t see your variable names or comments. With LLMs, we can write instructions the machine actually understands, enabling a new kind of human–computer collaboration.++**The abstraction level must be high enough to speak the language of each use case.** It should explicitly define concepts, include necessary explanations, and remove ambiguity. In short: self-documenting.++### Language fosters collaboration: users and agents building together++The language must be readable by everyone who matters: domain experts who know the business logic, engineers who optimize and deploy it, and crucially, AI agents that can build and refine workflows autonomously.++Imagine agents that transform natural language requirements into working workflows. They design each transformation step (or reuse existing ones), test against real or synthetic data, incorporate expert feedback, and iterate to improve quality while reducing costs. Once a workflow is built, agents can invoke it as a reliable tool whenever they need structured, predictable outputs.++> This is how agents finally remember know-how: by encoding methods into reusable workflows they can build, share, and execute on demand.+>++## The Know-How Graph: a Network of Composable Methods++**Breaking complex work into smaller tasks is a recursive, core pattern.** Each workflow should stand on the shoulders of others, composing like LEGO bricks to build increasingly sophisticated cognitive systems.++What emerges is a **Know-How Graph**: not just static knowledge, but executable methods that connect and build upon one another. **Unlike a knowledge graph mapping facts, this maps procedures: the actual know-how of getting cognitive work done.**++**Example:**++A recruitment workflow doesn't start from scratch. It composes existing workflows:++- ExtractCandidateProfile (experience, education, skills…)+- ExtractJobOffer (skills, years of experience…).++These feed into your custom ScoreCard logic to produce a MatchAnalysis, which triggers either GenerateRejectionEmail or PrepareInterviewQuestions.++Each component can be assigned to different team members and validated independently by the relevant stakeholders.++> Think of a workflow as a proven route through the work, and the Know-How Graph as the network of all such routes.+>++### Know-how is as shareable as knowledge++Think about the explosion of prompt sharing since 2023. All those people trading their best ChatGPT prompts on Twitter, GitHub, Reddit, LinkedIn. Now imagine that same viral knowledge sharing, but with complete, tested, composable workflows instead of fragile prompts.++We’ve seen this movie: software package managers, SQL views, Docker, dbt packages. Composable standards create ecosystems where everyone’s work makes everyone else more productive. Generic workflows for common tasks will spread rapidly, while companies keep their differentiating workflows as competitive advantage. That's how we stop reinventing the wheel while preserving secret sauce.++The same principle applies to AI workflows through the Know-How Graph: durable infrastructure that compounds value over time.++> The Know-How Graph will thrive on the open web because workflows are just files: easy to publish, fork, improve, and compose.+>++### What this unlocks++- Faster time to production (reuse existing workflows + AI writes them for you)+- Lower run costs (optimize price / performance for each task)+- Better collaboration between tech and business+- Better auditability / compliance+- No vendor lock-in++## Our Solution: Pipelex++[**Pipelex**](https://github.com/Pipelex/pipelex) is our take on this language: open-source (MIT), designed for the Know-How Graph.++Each workflow is built from pipes: modular transformations that guarantee their output structure while applying intelligence to the content. A pipe is a knowledge transformer with a simple contract: knowledge in → knowledge out., each defined conceptually and with explicit structure and validation. The method is readable and editable by humans and agents.++Our Pipelex workflow builder is itself a Pipelex workflow. The tooling builds itself.++## Why This Can Become a Standard++Pipelex is MIT-licensed and designed for portability. Workflows are files, based on TOML syntax (itself well standardized), and the outputs are validated JSON.++Early adopters are contributing to the [cookbook repo](https://github.com/Pipelex/pipelex-cookbook), building integrations, and running workflows in production. The pieces for ecosystem growth are in place: declarative spec, reference implementation, composable architecture.++Building a standard is hard. We're at v0.1.0, with versioning and backward compatibility coming next. The spec will evolve with your feedback.++## Join Us++The most valuable standards are boring infrastructure everyone relies on: SQL, HTTP, JSON. Pipelex aims to be that for AI workflows.++Start with one workflow: extract invoice data, process applications, analyze reports… Share what works. Build on what others share.++**The future of AI needs both:** smarter agents that explore and adapt, AND reliable workflows that execute proven methods at scale. One workflow at a time, let's build the cognitive infrastructure every organization needs.++---++© 2025 Evotis S.A.S.+Pipelex is a trademark of Evotis S.A.S.\ No newline at end of filediff --git a/docs/index.md b/docs/index.mdindex a83d9b5bf..095fe73cd 100644--- a/docs/index.md+++ b/docs/index.md@@ -56,7 +56,7 @@ Pipelex solves these problems by:Ready to get started?- [:material-rocket-launch: Get Started](./home/2-get-started/pipe-builder.md){ .md-button .md-button--primary }-- [:material-book-open: Read the Manifesto](./home/3-understand-pipelex/manifesto.md){ .md-button }+- [:material-book-open: Read the Know-How Graph](./home/3-understand-pipelex/viewpoint.md){ .md-button }- [:material-lightbulb: Explore the Paradigm](./home/3-understand-pipelex/pipelex-paradigm/index.md){ .md-button }**Explore Examples:**diff --git a/mkdocs.yml b/mkdocs.ymlindex bef311f35..75b90f250 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -93,8 +93,9 @@ nav:- The Pipe Builder: home/2-get-started/pipe-builder.md- Write Workflows Manually: home/2-get-started/write-workflows-manually.md- Understand Pipelex:- - The Knowledge Pipeline Manifesto: home/3-understand-pipelex/manifesto.md+ - The Know-How Graph: home/3-understand-pipelex/viewpoint.md- The Pipelex Paradigm: home/3-understand-pipelex/pipelex-paradigm/index.md+ - Pipelex Language Spec v0.1.0: home/3-understand-pipelex/language-spec-v0-1-0.md- Cookbook Examples:- Overview: home/4-cookbook-examples/index.md- Hello World: home/4-cookbook-examples/hello-world.md