Skip to content

The only tool you need to know what is happening and how to fix it.

16.5 KB
Raw Blame
# Embedded Mode

import { Callout, Steps } from "nextra/components";

Embedded mode runs a full Traceway server inside your Go process. Users, projects and settings live in SQLite, and telemetry lives in **DuckDB**. Both run in your process, in memory or as local files. No Docker, no infrastructure: just one `tracewaybackend.Run()` call and your app has local observability.

This is intended for **local development only**. For production, use the [Docker Compose](/server/docker-compose) or [minimal](/server/minimal) deployment.

<Callout type="warning">
  **Build your app with `-tags telemetry_duckdb`.** The tag selects the DuckDB telemetry store, which is the one we recommend and the one this page documents. A build without the tag still works, but it falls back to SQLite telemetry, which is being phased out. See [Builds without the tag](#builds-without-the-tag).
</Callout>

<Callout type="info">
  **Want to see it in action?** Check out the [full working example on
  GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel):
  a complete Go app with embedded Traceway and OpenTelemetry instrumentation
  you can run in one command.
</Callout>

## Quick Start

<Steps>

### Install the Traceway backend module

```bash
go get github.com/tracewayapp/traceway/backend
```

<Callout type="warning">
  **On backend v1.9.15 and earlier, that command fails.** Those releases declare
  their dependency on the Traceway CLI module with a placeholder version, so
  `go get` stops with:

  ```
  github.com/tracewayapp/traceway/[email protected]: invalid version: unknown revision 000000000000
  ```

  Point that dependency at a real published CLI version in your own `go.mod`:

  ```
  replace github.com/tracewayapp/traceway/cli => github.com/tracewayapp/traceway/cli v1.9.15
  ```

  Then run `go get github.com/tracewayapp/traceway/backend` again. The failed
  attempt leaves your `go.mod` untouched, so there is nothing to clean up first.
  The CLI and the backend are released together, so use the CLI version that
  matches the backend version you want. A `replace` only applies to your own
  module, so it does not affect anything that imports your code.

  Later releases need none of this. Upgrade instead if you can.
</Callout>

### Start Traceway inside your app

Call `tracewaybackend.Run()` in a goroutine during startup. It boots an in-memory server and seeds a default user and project:

```go
package main

import (
	"github.com/gin-gonic/gin"
	tracewaybackend "github.com/tracewayapp/traceway/backend"
)

func main() {
	go tracewaybackend.Run(
		tracewaybackend.WithPort(8082),
		tracewaybackend.WithDefaultUser("[email protected]", "admin"),
		tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
	)

	router := gin.Default()
	// ... your routes
	router.Run(":8080")
}
```

### Point OpenTelemetry to your local Traceway

Install the OTel dependencies:

```bash
go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
       go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
```

Configure the OTLP exporter to send traces to the embedded server, and add the OTel middleware to your router:

```go
package main

import (
	"context"
	"time"

	"github.com/gin-gonic/gin"
	tracewaybackend "github.com/tracewayapp/traceway/backend"
	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

const tracewayToken = "dev-token"

func main() {
	go tracewaybackend.Run(
		tracewaybackend.WithPort(8082),
		tracewaybackend.WithDefaultUser("[email protected]", "admin"),
		tracewaybackend.WithDefaultProject("My App", "opentelemetry", tracewayToken),
	)

	shutdown := initTracer()
	defer shutdown()

	router := gin.Default()
	router.Use(otelgin.Middleware("my-app"))

	router.GET("/ping", func(c *gin.Context) {
		c.String(200, "pong")
	})
	router.Run(":8080")
}

func initTracer() func() {
	exporter, err := otlptracehttp.New(
		context.Background(),
		otlptracehttp.WithEndpointURL("http://localhost:8082/api/otel/v1/traces"),
		otlptracehttp.WithHeaders(map[string]string{
			"Authorization": "Bearer " + tracewayToken,
		}),
	)
	if err != nil {
		panic(err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("my-app"),
		)),
	)
	otel.SetTracerProvider(tp)

	// Go's global propagator is a no-op by default. Without this line the
	// traceparent header is never sent or read, so an outgoing call starts a
	// brand new trace instead of joining the request it came from.
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	return func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		tp.Shutdown(ctx)
	}
}
```

The key parts:
- **Exporter URL** points to `http://localhost:8082/api/otel/v1/traces`, the embedded server's OTLP endpoint
- **Authorization header** uses the same token from `WithDefaultProject`
- **`otel.SetTextMapPropagator`** is required in Go. The global propagator ships as a no-op, so without this call an HTTP call your app makes to another service shows up as its own separate trace instead of a child span
- **`defer shutdown()`** flushes the batch processor on exit. Without it the last few seconds of spans never leave the process

### Run your app with the DuckDB tag

```bash
go run -tags telemetry_duckdb .
```

The tag needs CGO, so a C compiler has to be installed. [DuckDB telemetry](#duckdb-telemetry) below covers the requirements.

The startup log tells you which store you got. This is the line to look for:

```
DuckDB telemetry database opened at
```

A path follows it once you keep data on disk. If the line says `SQLite telemetry database opened` instead, the tag did not reach the build.

### Open the dashboard

Open **[http://localhost:8082](http://localhost:8082)** in your browser.

Log in with the credentials from `WithDefaultUser` (`[email protected]` / `admin`).

Hit your app at `http://localhost:8080/ping` a few times. Traces will appear in the Traceway dashboard within a few seconds.

</Steps>

## DuckDB telemetry

DuckDB is a columnar engine that runs inside your process, the same way SQLite does. Traceway stores traces, issues, logs and metrics in it because dashboard queries stay fast as data grows.

### Build requirements

- **The build tag.** Pass `-tags telemetry_duckdb` to every `go build`, `go run` and `go test` of your app, including the ones your IDE, live-reload tool or Dockerfile run. To set it once, export `GOFLAGS=-tags=telemetry_duckdb`.
- **CGO and a C compiler.** DuckDB is linked as a native library. CGO is on by default when a compiler is installed: the Xcode command line tools on macOS, `gcc` on Linux. If you turned it off, set `CGO_ENABLED=1`.
- **A supported platform.** The driver ships prebuilt libraries for macOS (Intel and Apple silicon), Linux with glibc (amd64 and arm64) and Windows (amd64). Alpine and other musl-based images are not supported, so use a Debian or Ubuntu base for a dev container.

There is nothing extra to `go get`: the DuckDB driver is already a dependency of the backend module. Expect the binary to grow by about 50 MB.

If CGO is off, the build stops with an error like this:

```
build constraints exclude all Go files in .../github.com/duckdb/duckdb-go-bindings/lib/[email protected]
```

Install a C compiler, make sure `CGO_ENABLED` is not `0`, and build again. Cross-compiling needs a C toolchain for the target, so the simplest route is to build on the platform you run on.

### Settings

The embedded server reads these from your process environment. Set them before `Run` is called.

| Variable | Default | Description |
| --- | --- | --- |
| `DUCKDB_MEMORY_LIMIT` | *(unset, about 80% of the RAM DuckDB sees)* | Memory budget for the engine, for example `1GB`. A ceiling, not a reservation. |
| `DUCKDB_THREADS` | *(unset, all cores)* | Caps the engine's thread count, for example `2`. |
| `DUCKDB_CHECKPOINT_THRESHOLD` | *(unset, 16MB)* | WAL size that triggers a checkpoint, for example `256MB`. Rarely needs changing in development. |
| `DUCKDB_RETENTION_DAYS` | `30` | Days of telemetry to keep. A cleanup pass runs at startup and then every hour. Set to `0` to keep everything. |

On a laptop the defaults are fine. Inside a memory-capped dev container DuckDB sees the host's RAM, not the container limit, so set `DUCKDB_MEMORY_LIMIT` to about half the container's memory. A value DuckDB cannot parse makes `Run` panic at startup with a message that names the option.

<Callout type="info">
  Backend v2.0.3 and earlier ignore these four variables in embedded mode and always use the defaults.
</Callout>

## Configuration Options

All options are passed to `tracewaybackend.Run()`. When any options are provided, embedded mode is activated automatically.

Embedded mode signs dashboard sessions with the key from `WithJWTSecret`, else from the `TRACEWAY_JWT_SECRET` environment variable. It never reads `JWT_SECRET`, because that name usually belongs to your own application. With neither set, each start generates a random signing key and logs a line saying so, and dashboard sessions expire on restart. To keep sessions across restarts with a persistent database, pass a stable, unique secret of at least 32 characters. An empty, short, or former built-in secret stops startup and exits the process.

| Option                                              | Description                                                                                        | Default                   |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------- |
| `WithPort(port)`                                    | HTTP port for the Traceway server                                                                  | `8082`                    |
| `WithServerURL(url)`                                | Base URL override (used in connection string output)                                               | `http://localhost:{port}` |
| `WithSQLitePath(path)`                              | Path of the main SQLite database file. The telemetry file is created next to it                    | `:memory:` (in-memory)    |
| `WithDefaultUser(email, password)`                  | Create a default admin user on startup                                                             | None                      |
| `WithDefaultProject(name, framework, token)`        | Create a default project with a fixed token (can be called multiple times)                         | None                      |
| `DisableLogging()`                                  | Silence the embedded server's startup and request logs so they don't mix with your app's output    | Logging enabled           |
| `WithDefaultProjectSourceMapToken(name, token)`     | Set a fixed source map upload token on a default project. Call it after the matching `WithDefaultProject`, since it looks the project up by name | None |
| `WithMonitoringURL(connectionString)`               | Report the embedded server's own telemetry to another Traceway instance. Takes a connection string in the form `<project_token>@<url>/api/report` | None |
| `WithJWTSecret(secret)`                             | Signing key for dashboard sessions, at least 32 characters. Overrides `TRACEWAY_JWT_SECRET`         | Random key per start      |

If the embedded server's startup lines get in the way of your own output, add `tracewaybackend.DisableLogging()` to the option list:

```go
go tracewaybackend.Run(
	tracewaybackend.WithPort(8082),
	tracewaybackend.DisableLogging(),
	tracewaybackend.WithDefaultUser("[email protected]", "admin"),
	tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
```

## Multiple Projects

`WithDefaultProject` can be called multiple times to create separate projects: one for your Go backend, another for a frontend or any other service. Each project gets its own token, keeping telemetry organized in the dashboard:

```go
go tracewaybackend.Run(
	tracewaybackend.WithPort(8082),
	tracewaybackend.WithDefaultUser("[email protected]", "admin"),
	tracewaybackend.WithDefaultProject("Backend", "opentelemetry", "backend-token"),
	tracewaybackend.WithDefaultProject("Frontend", "svelte", "frontend-token"),
)
```

## Data Persistence

By default, both databases are **in-memory**, which means all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:

Add `WithSQLitePath` to the options you already pass. It does not replace them. Keep the `go` keyword too, otherwise `Run` blocks and your own server never starts.

```go
go tracewaybackend.Run(
	tracewaybackend.WithPort(8082),
	tracewaybackend.WithSQLitePath("./traceway.db"),
	tracewaybackend.WithDefaultUser("[email protected]", "admin"),
	tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
)
```

### What ends up on disk

`WithSQLitePath("./traceway.db")` creates **two** databases side by side, not one:

| File | Contents |
| --- | --- |
| `traceway.db` | SQLite. Relational data: users, organizations, projects, settings |
| `traceway_telemetry.duckdb` | DuckDB. Telemetry: traces, endpoints, issues, logs, metrics |

Both write sidecar files: `traceway.db-wal` and `traceway.db-shm` for SQLite, `traceway_telemetry.duckdb.wal` for DuckDB. Back up or ignore all of them, not just the file you named. To copy the DuckDB file safely, stop your app first and take the `.wal` file with it.

The server also creates a `storage/` directory in the process working directory for session recordings and synthetics screenshots. That directory appears in the default in-memory mode too.

Only one process can open the DuckDB file at a time. A second copy of your app pointed at the same path panics at startup with `Could not set lock on file`.

Add all of it to `.gitignore`:

```text
traceway.db*
traceway_telemetry.duckdb*
storage/
```

Seeding is idempotent: if the user from `WithDefaultUser` already exists, it's skipped.

## Builds without the tag

A plain `go build` still works. It stores telemetry in SQLite instead of DuckDB, in `traceway_telemetry.db` with `-wal` and `-shm` sidecar files. SQLite telemetry is being phased out, so use it only where you cannot enable CGO.

Two things to know while you are on it:

- **Memory.** A file-backed SQLite telemetry database keeps a page cache inside your process, and its default can grow to several gigabytes. Set the `SQLITE_CACHE_SIZE_MB` environment variable, for example `16`, before your app starts. [Memory sizing](/server/sqlite#memory-sizing) explains the numbers. The in-memory default is not affected.
- **Switching to DuckDB.** Add the tag and rebuild. Users, projects and tokens carry over, because the main database is the same file. Telemetry does not: the DuckDB build starts a fresh `traceway_telemetry.duckdb`, and the old `traceway_telemetry.db` files can be deleted.

See [Build Tags](/learn/build-tags) for the full list of storage combinations.

## Development-Only Guard

A common pattern is guarding embedded mode behind an environment check so it only runs in development:

```go
if os.Getenv("ENV") != "production" {
	go tracewaybackend.Run(
		tracewaybackend.WithPort(8082),
		tracewaybackend.WithDefaultUser("[email protected]", "admin"),
		tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),
	)
}
```

## Full Example

For a complete working example with error recording and child spans, see the [embedded-backend-otel example on GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel).

## Next Steps

The Quick Start above only sends request traces. Everything else works the same against the embedded server as against a hosted one, so point these guides at `http://localhost:8082` and use your `WithDefaultProject` token:

- [Traces](/client/otel/traces): child spans, recording errors so they land in Issues with a stack trace, and background jobs. A background job span needs `SpanKind.CONSUMER`, a plain root span is silently dropped.
- [Logs](/client/otel/logs): emitting logs that attach to the request's trace.
- [Metrics](/client/otel/metrics): counters, histograms, and how Traceway stores them.