The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
# Embedded Mode
import { Callout, Steps } from "nextra/components";
Embedded mode runs a full Traceway server inside your Go process. Users, projects and settings live in SQLite, and telemetry lives in **DuckDB**. Both run in your process, in memory or as local files. No Docker, no infrastructure: just one `tracewaybackend.Run()` call and your app has local observability.
This is intended for **local development only**. For production, use the [Docker Compose](/server/docker-compose) or [minimal](/server/minimal) deployment.
<Callout type="warning">
**Build your app with `-tags telemetry_duckdb`.** The tag selects the DuckDB telemetry store, which is the one we recommend and the one this page documents. A build without the tag still works, but it falls back to SQLite telemetry, which is being phased out. See [Builds without the tag](#builds-without-the-tag).
</Callout>
<Callout type="info">
**Want to see it in action?** Check out the [full working example on
GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel):
a complete Go app with embedded Traceway and OpenTelemetry instrumentation
you can run in one command.
</Callout>
## Quick Start
<Steps>
### Install the Traceway backend module
```bash
go get github.com/tracewayapp/traceway/backend
```
<Callout type="warning">
**On backend v1.9.15 and earlier, that command fails.** Those releases declare
their dependency on the Traceway CLI module with a placeholder version, so
`go get` stops with:
```
github.com/tracewayapp/traceway/[email protected]: invalid version: unknown revision 000000000000
```
Point that dependency at a real published CLI version in your own `go.mod`:
```
replace github.com/tracewayapp/traceway/cli => github.com/tracewayapp/traceway/cli v1.9.15
```
Then run `go get github.com/tracewayapp/traceway/backend` again. The failed
attempt leaves your `go.mod` untouched, so there is nothing to clean up first.
The CLI and the backend are released together, so use the CLI version that
matches the backend version you want. A `replace` only applies to your own
module, so it does not affect anything that imports your code.
Later releases need none of this. Upgrade instead if you can.
</Callout>
### Start Traceway inside your app
Call `tracewaybackend.Run()` in a goroutine during startup. It boots an in-memory server and seeds a default user and project:
```go
package main
import (
"github.com/gin-gonic/gin"
tracewaybackend "github.com/tracewayapp/traceway/backend"
)
func main() {
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
router := gin.Default()
// ... your routes
router.Run(":8080")
}
```
### Point OpenTelemetry to your local Traceway
Install the OTel dependencies:
```bash
go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
```
Configure the OTLP exporter to send traces to the embedded server, and add the OTel middleware to your router:
```go
package main
import (
"context"
"time"
"github.com/gin-gonic/gin"
tracewaybackend "github.com/tracewayapp/traceway/backend"
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
const tracewayToken = "dev-token"
func main() {
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("My App", "opentelemetry", tracewayToken),
)
shutdown := initTracer()
defer shutdown()
router := gin.Default()
router.Use(otelgin.Middleware("my-app"))
router.GET("/ping", func(c *gin.Context) {
c.String(200, "pong")
})
router.Run(":8080")
}
func initTracer() func() {
exporter, err := otlptracehttp.New(
context.Background(),
otlptracehttp.WithEndpointURL("http://localhost:8082/api/otel/v1/traces"),
otlptracehttp.WithHeaders(map[string]string{
"Authorization": "Bearer " + tracewayToken,
}),
)
if err != nil {
panic(err)
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName("my-app"),
)),
)
otel.SetTracerProvider(tp)
// Go's global propagator is a no-op by default. Without this line the
// traceparent header is never sent or read, so an outgoing call starts a
// brand new trace instead of joining the request it came from.
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
return func() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
tp.Shutdown(ctx)
}
}
```
The key parts:
- **Exporter URL** points to `http://localhost:8082/api/otel/v1/traces`, the embedded server's OTLP endpoint
- **Authorization header** uses the same token from `WithDefaultProject`
- **`otel.SetTextMapPropagator`** is required in Go. The global propagator ships as a no-op, so without this call an HTTP call your app makes to another service shows up as its own separate trace instead of a child span
- **`defer shutdown()`** flushes the batch processor on exit. Without it the last few seconds of spans never leave the process
### Run your app with the DuckDB tag
```bash
go run -tags telemetry_duckdb .
```
The tag needs CGO, so a C compiler has to be installed. [DuckDB telemetry](#duckdb-telemetry) below covers the requirements.
The startup log tells you which store you got. This is the line to look for:
```
DuckDB telemetry database opened at
```
A path follows it once you keep data on disk. If the line says `SQLite telemetry database opened` instead, the tag did not reach the build.
### Open the dashboard
Open **[http://localhost:8082](http://localhost:8082)** in your browser.
Log in with the credentials from `WithDefaultUser` (`[email protected]` / `admin`).
Hit your app at `http://localhost:8080/ping` a few times. Traces will appear in the Traceway dashboard within a few seconds.
</Steps>
## DuckDB telemetry
DuckDB is a columnar engine that runs inside your process, the same way SQLite does. Traceway stores traces, issues, logs and metrics in it because dashboard queries stay fast as data grows.
### Build requirements
- **The build tag.** Pass `-tags telemetry_duckdb` to every `go build`, `go run` and `go test` of your app, including the ones your IDE, live-reload tool or Dockerfile run. To set it once, export `GOFLAGS=-tags=telemetry_duckdb`.
- **CGO and a C compiler.** DuckDB is linked as a native library. CGO is on by default when a compiler is installed: the Xcode command line tools on macOS, `gcc` on Linux. If you turned it off, set `CGO_ENABLED=1`.
- **A supported platform.** The driver ships prebuilt libraries for macOS (Intel and Apple silicon), Linux with glibc (amd64 and arm64) and Windows (amd64). Alpine and other musl-based images are not supported, so use a Debian or Ubuntu base for a dev container.
There is nothing extra to `go get`: the DuckDB driver is already a dependency of the backend module. Expect the binary to grow by about 50 MB.
If CGO is off, the build stops with an error like this:
```
build constraints exclude all Go files in .../github.com/duckdb/duckdb-go-bindings/lib/[email protected]
```
Install a C compiler, make sure `CGO_ENABLED` is not `0`, and build again. Cross-compiling needs a C toolchain for the target, so the simplest route is to build on the platform you run on.
### Settings
The embedded server reads these from your process environment. Set them before `Run` is called.
| Variable | Default | Description |
| --- | --- | --- |
| `DUCKDB_MEMORY_LIMIT` | *(unset, about 80% of the RAM DuckDB sees)* | Memory budget for the engine, for example `1GB`. A ceiling, not a reservation. |
| `DUCKDB_THREADS` | *(unset, all cores)* | Caps the engine's thread count, for example `2`. |
| `DUCKDB_CHECKPOINT_THRESHOLD` | *(unset, 16MB)* | WAL size that triggers a checkpoint, for example `256MB`. Rarely needs changing in development. |
| `DUCKDB_RETENTION_DAYS` | `30` | Days of telemetry to keep. A cleanup pass runs at startup and then every hour. Set to `0` to keep everything. |
On a laptop the defaults are fine. Inside a memory-capped dev container DuckDB sees the host's RAM, not the container limit, so set `DUCKDB_MEMORY_LIMIT` to about half the container's memory. A value DuckDB cannot parse makes `Run` panic at startup with a message that names the option.
<Callout type="info">
Backend v2.0.3 and earlier ignore these four variables in embedded mode and always use the defaults.
</Callout>
## Configuration Options
All options are passed to `tracewaybackend.Run()`. When any options are provided, embedded mode is activated automatically.
Embedded mode signs dashboard sessions with the key from `WithJWTSecret`, else from the `TRACEWAY_JWT_SECRET` environment variable. It never reads `JWT_SECRET`, because that name usually belongs to your own application. With neither set, each start generates a random signing key and logs a line saying so, and dashboard sessions expire on restart. To keep sessions across restarts with a persistent database, pass a stable, unique secret of at least 32 characters. An empty, short, or former built-in secret stops startup and exits the process.
| Option | Description | Default |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------- |
| `WithPort(port)` | HTTP port for the Traceway server | `8082` |
| `WithServerURL(url)` | Base URL override (used in connection string output) | `http://localhost:{port}` |
| `WithSQLitePath(path)` | Path of the main SQLite database file. The telemetry file is created next to it | `:memory:` (in-memory) |
| `WithDefaultUser(email, password)` | Create a default admin user on startup | None |
| `WithDefaultProject(name, framework, token)` | Create a default project with a fixed token (can be called multiple times) | None |
| `DisableLogging()` | Silence the embedded server's startup and request logs so they don't mix with your app's output | Logging enabled |
| `WithDefaultProjectSourceMapToken(name, token)` | Set a fixed source map upload token on a default project. Call it after the matching `WithDefaultProject`, since it looks the project up by name | None |
| `WithMonitoringURL(connectionString)` | Report the embedded server's own telemetry to another Traceway instance. Takes a connection string in the form `<project_token>@<url>/api/report` | None |
| `WithJWTSecret(secret)` | Signing key for dashboard sessions, at least 32 characters. Overrides `TRACEWAY_JWT_SECRET` | Random key per start |
If the embedded server's startup lines get in the way of your own output, add `tracewaybackend.DisableLogging()` to the option list:
```go
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.DisableLogging(),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
```
## Multiple Projects
`WithDefaultProject` can be called multiple times to create separate projects: one for your Go backend, another for a frontend or any other service. Each project gets its own token, keeping telemetry organized in the dashboard:
```go
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("Backend", "opentelemetry", "backend-token"),
tracewaybackend.WithDefaultProject("Frontend", "svelte", "frontend-token"),
)
```
## Data Persistence
By default, both databases are **in-memory**, which means all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:
Add `WithSQLitePath` to the options you already pass. It does not replace them. Keep the `go` keyword too, otherwise `Run` blocks and your own server never starts.
```go
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.WithSQLitePath("./traceway.db"),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
```
### What ends up on disk
`WithSQLitePath("./traceway.db")` creates **two** databases side by side, not one:
| File | Contents |
| --- | --- |
| `traceway.db` | SQLite. Relational data: users, organizations, projects, settings |
| `traceway_telemetry.duckdb` | DuckDB. Telemetry: traces, endpoints, issues, logs, metrics |
Both write sidecar files: `traceway.db-wal` and `traceway.db-shm` for SQLite, `traceway_telemetry.duckdb.wal` for DuckDB. Back up or ignore all of them, not just the file you named. To copy the DuckDB file safely, stop your app first and take the `.wal` file with it.
The server also creates a `storage/` directory in the process working directory for session recordings and synthetics screenshots. That directory appears in the default in-memory mode too.
Only one process can open the DuckDB file at a time. A second copy of your app pointed at the same path panics at startup with `Could not set lock on file`.
Add all of it to `.gitignore`:
```text
traceway.db*
traceway_telemetry.duckdb*
storage/
```
Seeding is idempotent: if the user from `WithDefaultUser` already exists, it's skipped.
## Builds without the tag
A plain `go build` still works. It stores telemetry in SQLite instead of DuckDB, in `traceway_telemetry.db` with `-wal` and `-shm` sidecar files. SQLite telemetry is being phased out, so use it only where you cannot enable CGO.
Two things to know while you are on it:
- **Memory.** A file-backed SQLite telemetry database keeps a page cache inside your process, and its default can grow to several gigabytes. Set the `SQLITE_CACHE_SIZE_MB` environment variable, for example `16`, before your app starts. [Memory sizing](/server/sqlite#memory-sizing) explains the numbers. The in-memory default is not affected.
- **Switching to DuckDB.** Add the tag and rebuild. Users, projects and tokens carry over, because the main database is the same file. Telemetry does not: the DuckDB build starts a fresh `traceway_telemetry.duckdb`, and the old `traceway_telemetry.db` files can be deleted.
See [Build Tags](/learn/build-tags) for the full list of storage combinations.
## Development-Only Guard
A common pattern is guarding embedded mode behind an environment check so it only runs in development:
```go
if os.Getenv("ENV") != "production" {
go tracewaybackend.Run(
tracewaybackend.WithPort(8082),
tracewaybackend.WithDefaultUser("[email protected]", "admin"),
tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
}
```
## Full Example
For a complete working example with error recording and child spans, see the [embedded-backend-otel example on GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel).
## Next Steps
The Quick Start above only sends request traces. Everything else works the same against the embedded server as against a hosted one, so point these guides at `http://localhost:8082` and use your `WithDefaultProject` token:
- [Traces](/client/otel/traces): child spans, recording errors so they land in Issues with a stack trace, and background jobs. A background job span needs `SpanKind.CONSUMER`, a plain root span is silently dropped.
- [Logs](/client/otel/logs): emitting logs that attach to the request's trace.
- [Metrics](/client/otel/metrics): counters, histograms, and how Traceway stores them.