Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Documentation: clarified (#514)

Louis Choquel8e952fc192a6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ce5b99cc9..957ecba91 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -48,6 +48,7 @@
- Telemetry system split into two streams:
1. Pipelex Gateway telemetry for service monitoring (never collects prompts/completions/business data)
2. Custom telemetry to user-configured backends
+- Documentation: clarified **Setup (first run)** vs **Configuration (TOML reference)**, added a Setup overview page, and added contributor docs for configuration defaults/overrides.
- `pipelex init` now creates a documented `telemetry.toml` template instead of prompting for preferences
- Model catalog updated with latest models (gpt-5.1, claude-4.5-opus, gemini-3.0-pro, etc.) and updated waterfalls in `base_deck.toml`
- Model constraints refactored from simple lists to structured `valued_constraints` dictionaries (e.g., `valued_constraints = { fixed_temperature = 1 }`)
diff --git a/docs/contribute/configuration-defaults-and-overrides.md b/docs/contribute/configuration-defaults-and-overrides.md
new file mode 100644
index 000000000..33eee645a
--- /dev/null
+++ b/docs/contribute/configuration-defaults-and-overrides.md
@@ -0,0 +1,50 @@
+# Configuration Internals (defaults and overrides)
+
+This page is for **contributors**: it explains where Pipelex defaults live, how configuration is merged at runtime, and where to change the behavior in code.
+
+For the user-facing configuration guide, see [Configuration Overview](../home/7-configuration/index.md).
+
+## Mental model (for contributors)
+
+Pipelex configuration is merged in layers:
+
+- **Shipped defaults**: maintained by the Pipelex project and used as the baseline by the installed package.
+- **Project overrides**: created in `.pipelex/` by `pipelex init …` and edited by projects that use Pipelex.
+- **Root override files** (advanced): optional files for machine/env/run-mode/super overrides.
+
+The key idea: client projects typically edit **`.pipelex/`**, while the Pipelex repository holds the **default values** used as a starting point.
+
+## Where defaults live (Pipelex repository)
+
+- **Default values** (baseline): repo root `pipelex.toml`
+- **User-facing templates** copied into `.pipelex/` by the init flow: `pipelex/kit/configs/…`
+
+## Merge order (runtime)
+
+The configuration loading/merging behavior is implemented in `pipelex/system/configuration/config_loader.py`.
+
+At a high level, the load order is:
+
+1. **Pipelex base defaults** (from the installed package)
+2. **Project config** from `.pipelex/pipelex.toml` (when running in a client project)
+3. **Root overrides** (optional), in this order:
+ 1. `pipelex_local.toml`
+ 2. `pipelex_{environment}.toml`
+ 3. `pipelex_{run_mode}.toml`
+ 4. `pipelex_super.toml`
+
+!!! note "Unit test special case"
+ When unit testing, run-mode overrides may be loaded from `tests/pipelex_{run_mode}.toml` (example: `tests/pipelex_unit_test.toml`).
+
+## Where to change things in code
+
+- **Config merge logic**: `pipelex/system/configuration/config_loader.py`
+- **Override lists** (`local` / `super`): `pipelex/system/configuration/config_root.py`
+
+## Contributor guidelines for config changes
+
+- If you change **defaults**, update the repo root `pipelex.toml` (and consider whether templates in `pipelex/kit/configs/…` should also be updated).
+- If you change **templates**, keep them user-focused and stable; avoid adding internal-only details.
+- If you change **merge order or override semantics**, treat it as a potentially breaking change and document it (changelog + migration notes if needed).
+
+
diff --git a/docs/home/5-setup/configure-ai-providers.md b/docs/home/5-setup/configure-ai-providers.md
index e7156e7db..4dace4584 100644
--- a/docs/home/5-setup/configure-ai-providers.md
+++ b/docs/home/5-setup/configure-ai-providers.md
@@ -36,7 +36,7 @@ That's it! Your pipelines can now access any supported LLM.
!!! info "Terms of Service & Telemetry"
When using Pipelex Gateway, you'll be prompted to accept our terms of service. By using the Gateway, identified telemetry is automatically enabled (tied to your hashed API key) to help us monitor service quality and enforce fair usage.
- **We collect only technical data** (model names, token counts, latency, error rates). We do **NOT** collect your prompts, completions, or business data. See our [Privacy Policy](https://go.pipelex.com/privacy-policy) for details.
+ **We collect only technical data** (model names, token counts, latency, error rates). We do **NOT** collect your prompts, completions, or business data. See [Telemetry](./telemetry.md) for details and trade-offs, and our [Privacy Policy](https://go.pipelex.com/privacy-policy) for more.
!!! note "Migration from pipelex_inference"
If you were using the deprecated `pipelex_inference` backend, migrate to `pipelex_gateway`:
diff --git a/docs/home/5-setup/index.md b/docs/home/5-setup/index.md
new file mode 100644
index 000000000..61a3cc0b7
--- /dev/null
+++ b/docs/home/5-setup/index.md
@@ -0,0 +1,19 @@
+# Setup (first run)
+
+This section is for **first-time onboarding**: the key choices you need to make to get a project running quickly and safely.
+
+If you already have a project running and want to tune behavior, jump to [Configuration (TOML reference)](../7-configuration/index.md).
+
+## What belongs where?
+
+- **Setup**: pick providers, create initial files, understand privacy/telemetry trade-offs, and follow recommended project layout.
+- **Configuration**: the detailed reference for each TOML section and its options.
+
+## Quick guide
+
+- **Need to run pipelines with LLMs?** Start with [Configure AI Providers](./configure-ai-providers.md).
+- **Need a recommended repo layout for `.plx` and Python code?** See [Project Organization](./project-organization.md).
+- **Need to understand telemetry and privacy trade-offs?** See [Telemetry](./telemetry.md).
+- **Ready to tune the knobs?** Go to [Configuration Overview](../7-configuration/index.md).
+
+
diff --git a/docs/home/5-setup/telemetry.md b/docs/home/5-setup/telemetry.md
index 23e7f947c..382540fdb 100644
--- a/docs/home/5-setup/telemetry.md
+++ b/docs/home/5-setup/telemetry.md
@@ -101,6 +101,6 @@ We take your privacy seriously:
- PostHog tracing includes privacy controls to redact sensitive content
- All telemetry can be completely disabled
-For detailed configuration options, see [Telemetry Configuration](../../home/7-configuration/config-practical/telemetry-config.md).
+For detailed configuration options, see [Telemetry Configuration](../7-configuration/config-practical/telemetry-config.md).
For more information about our data practices, see our [Privacy Policy](https://go.pipelex.com/privacy-policy).
diff --git a/docs/home/7-configuration/config-practical/telemetry-config.md b/docs/home/7-configuration/config-practical/telemetry-config.md
index 5ed8bd1b5..7e0bb1307 100644
--- a/docs/home/7-configuration/config-practical/telemetry-config.md
+++ b/docs/home/7-configuration/config-practical/telemetry-config.md
@@ -10,7 +10,7 @@ Telemetry configuration is stored in `.pipelex/telemetry.toml`. This file contro
1. **Gateway Telemetry** (Pipelex-controlled): Automatic when using Pipelex Gateway, tied to your API key
2. **Custom Telemetry** (User-controlled): Configured in this file, sent to your own backends
- For an overview, see [Telemetry Setup](../../../home/5-setup/telemetry.md).
+ For an overview, see [Telemetry Setup](../../5-setup/telemetry.md).
## Configuration File Location
diff --git a/docs/home/7-configuration/index.md b/docs/home/7-configuration/index.md
index d1140c6ee..6674978d3 100644
--- a/docs/home/7-configuration/index.md
+++ b/docs/home/7-configuration/index.md
@@ -2,7 +2,12 @@
## Overview
-Pipelex uses a TOML-based configuration system. The main configuration file `pipelex.toml` must be located at the root of your project. You can create this file by running:
+Pipelex uses a TOML-based configuration system with **shipped defaults** plus **project-level overrides**.
+
+- **Shipped defaults**: Pipelex ships default values that are maintained in the Pipelex repository (contributors will see them in the repo root `pipelex.toml`). This is the baseline used by the installed package.
+- **Project overrides**: a project that *uses* Pipelex typically customizes behavior via files created in `.pipelex/`.
+
+You can create the project configuration files by running:
```bash
pipelex init config
@@ -14,6 +19,26 @@ pipelex init config
For a complete list of all possible configuration options, refer to the configuration group documentation below.
+## Where to edit configuration in a project
+
+The main project configuration files are:
+
+- `.pipelex/pipelex.toml`: project customization (logging, reporting, tracker, feature flags, etc.)
+- `.pipelex/telemetry.toml`: custom telemetry destinations
+- `.pipelex/inference/…`: inference backends, routing profiles, and model presets
+
+## Overrides (advanced)
+
+In addition to `.pipelex/pipelex.toml`, Pipelex can apply override files at the **project root** (not in `.pipelex/`) for machine- and environment-specific settings:
+
+1. `pipelex_local.toml`
+2. `pipelex_{environment}.toml` (example: `pipelex_dev.toml`)
+3. `pipelex_{run_mode}.toml` (example: `pipelex_normal.toml`; unit tests may use `tests/pipelex_unit_test.toml`)
+4. `pipelex_super.toml` (recommended to gitignore)
+
+!!! info "Contributor details"
+ For the full “where defaults live” and “how config is merged” explanation, see [Configuration Internals](../../contribute/configuration-defaults-and-overrides.md).
+
## Configuration Structure
The configuration is organized into three main sections:
diff --git a/mkdocs.yml b/mkdocs.yml
index 5feb4b8ac..ccd3e752f 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -111,7 +111,8 @@ nav:
- Text Generation:
- Screenplay Generator (wip): home/4-cookbook-examples/write-screenplay.md
- Tweet Optimizer (wip): home/4-cookbook-examples/write-tweet.md
- - Setup:
+ - Setup (first run):
+ - Overview: home/5-setup/index.md
- Configure AI Providers: home/5-setup/configure-ai-providers.md
- Project Organization: home/5-setup/project-organization.md
- Telemetry: home/5-setup/telemetry.md
@@ -150,7 +151,7 @@ nav:
- Optimize Cost & Quality: home/6-build-reliable-ai-workflows/configure-ai-llm-to-optimize-workflows.md
- LLM Structured Generation: home/6-build-reliable-ai-workflows/llm-structured-generation-config.md
- LLM Prompting Style: home/6-build-reliable-ai-workflows/adapt-to-llm-prompting-style-openai-anthropic-mistral.md
- - Configuration:
+ - Configuration (TOML reference):
- Overview: home/7-configuration/index.md
- Pipeline Validation Configuration:
- Dry Run: home/7-configuration/config-pipeline-validation/dry-run-config.md
@@ -200,6 +201,7 @@ nav:
- n8n: https://pipelex.github.io/n8n-nodes-pipelex/
- Contribute:
- Contributing: contributing.md
+ - Configuration Internals: contribute/configuration-defaults-and-overrides.md
- Code of Conduct: CODE_OF_CONDUCT.md
- License: license.md
- Changelog: changelog.md