Skip to content

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search

fix: enable auto embedding by default for default inference configuration (#2976)

Mohamed Bassembcbb577664b7
diff --git a/apps/workers/workers/embeddingsWorker.ts b/apps/workers/workers/embeddingsWorker.ts
index 8899d90d..df7c4e11 100644
--- a/apps/workers/workers/embeddingsWorker.ts
+++ b/apps/workers/workers/embeddingsWorker.ts
@@ -12,7 +12,7 @@ import {
zEmbeddingsRequestSchema,
} from "@karakeep/shared-server";
import serverConfig from "@karakeep/shared/config";
-import { InferenceClientFactory } from "@karakeep/shared/inference";
+import { EmbeddingClientFactory } from "@karakeep/shared/inference";
import logger from "@karakeep/shared/logger";
import {
DequeuedJob,
@@ -425,10 +425,10 @@ async function runEmbed(
"user.id": bookmark.userId,
});
- const inferenceClient = InferenceClientFactory.build();
- if (!inferenceClient) {
+ const embeddingClient = EmbeddingClientFactory.build();
+ if (!embeddingClient) {
logger.debug(
- `[embeddings][${jobId}] No inference client configured, skipping embedding generation`,
+ `[embeddings][${jobId}] No embedding client configured, skipping embedding generation`,
);
if (shouldTag) {
await enqueueTagging(bookmarkId, bookmark.userId, job.priority);
@@ -454,7 +454,7 @@ async function runEmbed(
`[embeddings][${jobId}] Embedding text length: ${embeddingText.length} characters`,
);
- const embeddingResponse = await inferenceClient.generateEmbeddingFromText([
+ const embeddingResponse = await embeddingClient.generateEmbeddingFromText([
embeddingText,
]);
diff --git a/docs/docs/03-configuration/01-environment-variables.md b/docs/docs/03-configuration/01-environment-variables.md
index 911ef122..f1c9a704 100644
--- a/docs/docs/03-configuration/01-environment-variables.md
+++ b/docs/docs/03-configuration/01-environment-variables.md
@@ -106,7 +106,10 @@ Either `OPENAI_API_KEY` or `OLLAMA_BASE_URL` need to be set for automatic taggin
| INFERENCE_TEXT_MODEL | No | gpt-4.1-mini | The model to use for text inference. You'll need to change this to some other model if you're using ollama. |
| INFERENCE_IMAGE_MODEL | No | gpt-4o-mini | The model to use for image inference. You'll need to change this to some other model if you're using ollama and that model needs to support vision APIs (e.g. llava). |
| EMBEDDING_ENABLE_AUTO_INDEXING | No | enabled if using OpenAI | Whether bookmark embedding generation and vector indexing are enabled. This is enabled by default if you're using the default OpenAI settings (i.e. no Ollama, no OpenAI base URL override). Otherwise, you need to manually enable it after configuring the embedding model and its dimensions. |
+| EMBEDDING_OPENAI_API_KEY | No | `OPENAI_API_KEY` | The API key used for OpenAI-compatible embedding requests. If unset, embedding requests use `OPENAI_API_KEY`. |
+| EMBEDDING_OPENAI_BASE_URL | No | `OPENAI_BASE_URL` | The OpenAI-compatible API base URL used for embedding requests. If unset, embedding requests use `OPENAI_BASE_URL`. |
| EMBEDDING_TEXT_MODEL | No | text-embedding-3-small | The model to be used for generating embeddings for the text. |
+| EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE | No | Not set | Requests this output dimension from embedding models that support configurable dimensions. When set, it must match `EMBEDDING_DIMENSIONS`, or Karakeep will fail to start. |
| EMBEDDING_CONTEXT_LENGTH | No | 8000 | The max number of characters that will be passed to the embedding model. If the bookmark content is larger than this size, it will be truncated before embedding generation. |
| EMBEDDING_DIMENSIONS | No | 1536 | The embedding vector dimensions expected by the vector store. This must match the embedding model or provider configuration. |
| EMBEDDING_NUM_WORKERS | No | 1 | Number of concurrent workers for embedding generation and vector indexing tasks. Increase this if you have multiple embedding requests and want to process them in parallel. |
diff --git a/docs/docs/03-configuration/02-different-ai-providers.md b/docs/docs/03-configuration/02-different-ai-providers.md
index b46f4e5e..39f45ae2 100644
--- a/docs/docs/03-configuration/02-different-ai-providers.md
+++ b/docs/docs/03-configuration/02-different-ai-providers.md
@@ -1,8 +1,10 @@
# Configuring different AI Providers
-Karakeep uses LLM providers for AI tagging and summarization. We support OpenAI-compatible providers and ollama. This guide will show you how to configure different providers.
+Karakeep uses LLM providers for AI tagging and summarization. It also uses embedding models for stuff like semantic search. We support OpenAI-compatible providers and ollama. This guide will show you how to configure different providers.
-## OpenAI
+## Tagging and Summarization Models
+
+### OpenAI
If you want to use OpenAI itself, you just need to pass in the OPENAI_API_KEY environment variable.
@@ -14,9 +16,11 @@ OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# INFERENCE_IMAGE_MODEL=gpt-4o-mini
# EMBEDDING_TEXT_MODEL=text-embedding-3-small
# EMBEDDING_DIMENSIONS=1536
+# EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE=1536
```
-## Ollama
+
+### Ollama
Ollama is a local LLM provider that you can use to run your own LLM server. You'll need to pass ollama's address to karakeep and you need to ensure that it's accessible from within the karakeep container (e.g. no localhost addresses).
@@ -25,7 +29,7 @@ Ollama provides two API endpoints:
1. **OpenAI-compatible API (Recommended)** - Uses the `/v1` chat endpoint which handles message formatting automatically
2. **Native Ollama API** - Requires manual formatting for some models
-### Option 1: OpenAI-compatible API (Recommended)
+#### Option 1: OpenAI-compatible API (Recommended)
This approach uses Ollama's OpenAI-compatible endpoint and is more reliable with various models:
@@ -41,7 +45,7 @@ EMBEDDING_DIMENSIONS=768
EMBEDDING_CONTEXT_LENGTH=2048
```
-### Option 2: Native Ollama API
+#### Option 2: Native Ollama API
Alternatively, you can use the native Ollama API:
@@ -65,7 +69,7 @@ EMBEDDING_CONTEXT_LENGTH=2048
If you experience issues with certain models (especially OpenAI's gpt-oss models or other models requiring specific chat formats), try using the OpenAI-compatible API endpoint instead.
:::
-## Gemini
+### Gemini
Gemini has an OpenAI-compatible API. You need to get an api key from Google AI Studio. You also need to set up a billing account, even if using the free tier.
@@ -82,7 +86,7 @@ EMBEDDING_DIMENSIONS=3072
```
-## OpenRouter
+### OpenRouter
```
OPENAI_BASE_URL=https://openrouter.ai/api/v1
@@ -94,7 +98,7 @@ INFERENCE_IMAGE_MODEL=meta-llama/llama-4-scout
EMBEDDING_TEXT_MODEL=openai/text-embedding-3-small
```
-## Perplexity
+### Perplexity
```
OPENAI_BASE_URL=https://api.perplexity.ai
@@ -103,7 +107,7 @@ INFERENCE_TEXT_MODEL=sonar-pro
INFERENCE_IMAGE_MODEL=sonar-pro
```
-## Azure
+### Azure
Azure has an OpenAI-compatible API.
@@ -125,7 +129,7 @@ INFERENCE_TEXT_MODEL=YOUR_DEPLOYMENT_NAME
INFERENCE_IMAGE_MODEL=YOUR_DEPLOYMENT_NAME
```
-## Cloudflare
+### Cloudflare
Cloudflare supports OpenAI compatible endpoints. You can generate an API Token from the Cloudflare dashboard (Workers AI).
@@ -143,7 +147,7 @@ INFERENCE_OUTPUT_SCHEMA=json
```
-# Configuring Embedding Models
+## Embedding Models
Karakeep uses embedding models for stuff like semantic search and refining tag suggestions. Typically, when you configure an embedding model, you'll want to configure its default num dimensions and context length. Those are done via:
@@ -154,6 +158,20 @@ EMBEDDING_CONTEXT_LENGTH=
```
+Embeddings can use a different OpenAI-compatible provider. Set either override independently; any unset value falls back to its corresponding `OPENAI_*` setting.
+
+```
+EMBEDDING_OPENAI_API_KEY=embedding-provider-api-key
+EMBEDDING_OPENAI_BASE_URL=https://embedding-provider.example.com/v1
+```
+
+For embedding models that support multiple output sizes, set `EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE` to pass the requested dimensions to the provider. Its value must match `EMBEDDING_DIMENSIONS`, which configures the vector store, or Karakeep will fail to start:
+
+```
+EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE=768
+EMBEDDING_DIMENSIONS=768
+```
+
Once you've configured an embedding model, you'll also want to enable auto-embedding generation for your bookmarks by:
```
diff --git a/packages/shared/config.ts b/packages/shared/config.ts
index 0fad34a8..fd26f1b5 100644
--- a/packages/shared/config.ts
+++ b/packages/shared/config.ts
@@ -91,7 +91,14 @@ const allEnv = z.object({
INFERENCE_TEXT_MODEL: z.string().default("gpt-4.1-mini"),
INFERENCE_IMAGE_MODEL: z.string().default("gpt-4o-mini"),
EMBEDDING_ENABLE_AUTO_INDEXING: optionalStringBool(),
+ EMBEDDING_OPENAI_API_KEY: z.string().optional(),
+ EMBEDDING_OPENAI_BASE_URL: z.string().url().optional(),
EMBEDDING_TEXT_MODEL: z.string().default("text-embedding-3-small"),
+ EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE: z.coerce
+ .number()
+ .int()
+ .positive()
+ .optional(),
EMBEDDING_DIMENSIONS: z.coerce.number().default(1536),
EMBEDDING_CONTEXT_LENGTH: z.coerce.number().int().positive().default(8000),
EMBEDDING_NUM_WORKERS: z.coerce.number().default(1),
@@ -360,12 +367,19 @@ const serverConfigSchema = allEnv.transform((val, ctx) => {
semanticSearch: val.SEMANTIC_SEARCH_ENABLED,
},
embedding: {
+ isConfigured:
+ !!val.OPENAI_API_KEY ||
+ !!val.OLLAMA_BASE_URL ||
+ !!val.EMBEDDING_OPENAI_BASE_URL,
enableAutoIndexing:
val.EMBEDDING_ENABLE_AUTO_INDEXING === undefined
? // Enabled by default if using the default inference configuration (based on OpenAI models)
!val.OLLAMA_BASE_URL && !val.OPENAI_BASE_URL && !!val.OPENAI_API_KEY
: val.EMBEDDING_ENABLE_AUTO_INDEXING,
+ openAIApiKey: val.EMBEDDING_OPENAI_API_KEY,
+ openAIBaseUrl: val.EMBEDDING_OPENAI_BASE_URL,
textModel: val.EMBEDDING_TEXT_MODEL,
+ textModelDimensionOverride: val.EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE,
dimensions: val.EMBEDDING_DIMENSIONS,
contextLength: val.EMBEDDING_CONTEXT_LENGTH,
numWorkers: val.EMBEDDING_NUM_WORKERS,
@@ -548,6 +562,19 @@ const serverConfigSchema = allEnv.transform((val, ctx) => {
});
return z.NEVER;
}
+ if (
+ obj.embedding.textModelDimensionOverride !== undefined &&
+ obj.embedding.textModelDimensionOverride !== obj.embedding.dimensions
+ ) {
+ ctx.addIssue({
+ code: z.ZodIssueCode.custom,
+ message:
+ "EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE must match EMBEDDING_DIMENSIONS",
+ path: ["EMBEDDING_TEXT_MODEL_DIMENSION_OVERRIDE"],
+ fatal: true,
+ });
+ return z.NEVER;
+ }
return obj;
});
@@ -583,7 +610,7 @@ export const clientConfig = {
semanticSearchEnabled:
serverConfig.experimentalFeatures.semanticSearch &&
serverConfig.embedding.enableAutoIndexing &&
- serverConfig.inference.isConfigured,
+ serverConfig.embedding.isConfigured,
},
stripe: {
isConfigured: serverConfig.stripe.isConfigured,
diff --git a/packages/shared/inference.ts b/packages/shared/inference.ts
index 10ef73fd..57c176db 100644
--- a/packages/shared/inference.ts
+++ b/packages/shared/inference.ts
@@ -112,7 +112,11 @@ const defaultInferenceOptions: InferenceOptions = {
schema: null,
};
-export interface InferenceClient {
+export interface EmbeddingClient {
+ generateEmbeddingFromText(inputs: string[]): Promise<EmbeddingResponse>;
+}
+
+export interface InferenceClient extends EmbeddingClient {
inferFromText(
prompt: string,
opts: Partial<InferenceOptions>,
@@ -123,7 +127,6 @@ export interface InferenceClient {
image: string,
opts: Partial<InferenceOptions>,
): Promise<InferenceResponse>;
- generateEmbeddingFromText(inputs: string[]): Promise<EmbeddingResponse>;
}
const mapInferenceOutputSchema = <
@@ -169,6 +172,28 @@ export interface OpenAIInferenceConfig {
outputSchema: "structured" | "json" | "plain";
}
+export interface OpenAIEmbeddingConfig {
+ apiKey: string;
+ baseURL?: string;
+ proxyUrl?: string;
+ timeoutSec?: number;
+}
+
+const buildOpenAIClient = (config: OpenAIEmbeddingConfig) =>
+ new OpenAI({
+ apiKey: config.apiKey,
+ baseURL: config.baseURL,
+ timeout:
+ config.timeoutSec !== undefined ? config.timeoutSec * 1000 : undefined,
+ defaultHeaders: {
+ "X-Title": "Karakeep",
+ "HTTP-Referer": "https://karakeep.app",
+ },
+ fetchOptions: config.proxyUrl
+ ? { dispatcher: new undici.ProxyAgent(config.proxyUrl) }
+ : undefined,
+ });
+
export class InferenceClientFactory {
static build(): InferenceClient | null {
if (serverConfig.inference.openAIApiKey) {
@@ -182,6 +207,60 @@ export class InferenceClientFactory {
}
}
+export class EmbeddingClientFactory {
+ static build(): EmbeddingClient | null {
+ if (
+ serverConfig.embedding.openAIApiKey ||
+ serverConfig.embedding.openAIBaseUrl
+ ) {
+ const apiKey =
+ serverConfig.embedding.openAIApiKey ??
+ serverConfig.inference.openAIApiKey;
+ if (!apiKey) {
+ logger.error(
+ "EMBEDDING_OPENAI_API_KEY or OPENAI_API_KEY must be set when using EMBEDDING_OPENAI_BASE_URL",
+ );
+ return null;
+ }
+ return new OpenAIEmbeddingClient({
+ apiKey,
+ baseURL:
+ serverConfig.embedding.openAIBaseUrl ??
+ serverConfig.inference.openAIBaseUrl,
+ proxyUrl: serverConfig.inference.openAIProxyUrl,
+ timeoutSec: serverConfig.inference.openAITimeoutSec,
+ });
+ }
+
+ return InferenceClientFactory.build();
+ }
+}
+
+export class OpenAIEmbeddingClient implements EmbeddingClient {
+ private openAI: OpenAI;
+
+ constructor(config: OpenAIEmbeddingConfig) {
+ this.openAI = buildOpenAIClient(config);
+ }
+
+ async generateEmbeddingFromText(
+ inputs: string[],
+ ): Promise<EmbeddingResponse> {
+ const embedResponse = await this.openAI.embeddings.create({
+ model: serverConfig.embedding.textModel,
+ input: inputs,
+ ...(serverConfig.embedding.textModelDimensionOverride !== undefined
+ ? {
+ dimensions: serverConfig.embedding.textModelDimensionOverride,
+ }
+ : {}),
+ });
+ const embeddings = parseEmbeddingResponse(embedResponse);
+ const usage = parseEmbeddingUsage(embedResponse);
+ return { embeddings, ...usage };
+ }
+}
+
export class OpenAIInferenceClient implements InferenceClient {
openAI: OpenAI;
private config: OpenAIInferenceConfig;
@@ -189,19 +268,7 @@ export class OpenAIInferenceClient implements InferenceClient {
constructor(config: OpenAIInferenceConfig) {
this.config = config;
- this.openAI = new OpenAI({
- apiKey: config.apiKey,
- baseURL: config.baseURL,
- timeout:
- config.timeoutSec !== undefined ? config.timeoutSec * 1000 : undefined,
- defaultHeaders: {
- "X-Title": "Karakeep",
- "HTTP-Referer": "https://karakeep.app",
- },
- fetchOptions: config.proxyUrl
- ? { dispatcher: new undici.ProxyAgent(config.proxyUrl) }
- : undefined,
- });
+ this.openAI = buildOpenAIClient(config);
}
static fromConfig(): OpenAIInferenceClient {
@@ -315,6 +382,11 @@ export class OpenAIInferenceClient implements InferenceClient {
const embedResponse = await this.openAI.embeddings.create({
model: model,
input: inputs,
+ ...(serverConfig.embedding.textModelDimensionOverride !== undefined
+ ? {
+ dimensions: serverConfig.embedding.textModelDimensionOverride,
+ }
+ : {}),
});
const embedding2D = parseEmbeddingResponse(embedResponse);
const usage = parseEmbeddingUsage(embedResponse);
diff --git a/packages/trpc/routers/bookmarks.search.test.ts b/packages/trpc/routers/bookmarks.search.test.ts
index 16ba3ef2..3102faf3 100644
--- a/packages/trpc/routers/bookmarks.search.test.ts
+++ b/packages/trpc/routers/bookmarks.search.test.ts
@@ -10,7 +10,7 @@ import { createSemanticSearchRateLimitMiddleware } from "./bookmarks";
const searchMocks = vi.hoisted(() => ({
search: vi.fn(),
vectorSearch: vi.fn(),
- buildInferenceClient: vi.fn(),
+ buildEmbeddingClient: vi.fn(),
getVectorStoreClient: vi.fn(),
getRateLimitClient: vi.fn(),
checkRateLimit: vi.fn(),
@@ -20,8 +20,8 @@ const originalSemanticSearchFeature =
const originalEmbeddingAutoIndexing = serverConfig.embedding.enableAutoIndexing;
vi.mock("@karakeep/shared/inference", () => ({
- InferenceClientFactory: {
- build: searchMocks.buildInferenceClient,
+ EmbeddingClientFactory: {
+ build: searchMocks.buildEmbeddingClient,
},
}));
@@ -44,7 +44,7 @@ beforeEach<CustomTestContext>(async (context) => {
await defaultBeforeEach(true)(context);
searchMocks.search.mockReset();
searchMocks.vectorSearch.mockReset();
- searchMocks.buildInferenceClient.mockReset();
+ searchMocks.buildEmbeddingClient.mockReset();
searchMocks.getVectorStoreClient.mockReset();
searchMocks.getRateLimitClient.mockReset();
searchMocks.checkRateLimit.mockReset();
@@ -68,7 +68,7 @@ function mockEmbeddingInfra() {
promptTokens: 1,
totalTokens: 1,
}));
- searchMocks.buildInferenceClient.mockReturnValue({
+ searchMocks.buildEmbeddingClient.mockReturnValue({
generateEmbeddingFromText,
});
return generateEmbeddingFromText;
@@ -238,7 +238,7 @@ describe("bookmark search modes", () => {
type: BookmarkTypes.TEXT,
text: "fallback result",
});
- searchMocks.buildInferenceClient.mockReturnValue(null);
+ searchMocks.buildEmbeddingClient.mockReturnValue(null);
searchMocks.getVectorStoreClient.mockResolvedValue(null);
searchMocks.search.mockResolvedValue({
hits: [{ id: bookmark.id, score: 1 }],
@@ -392,7 +392,7 @@ describe("bookmark search modes", () => {
test<CustomTestContext>("rejects semantic search when its infrastructure is unavailable", async ({
apiCallers,
}) => {
- searchMocks.buildInferenceClient.mockReturnValue(null);
+ searchMocks.buildEmbeddingClient.mockReturnValue(null);
searchMocks.getVectorStoreClient.mockResolvedValue(null);
await expect(
diff --git a/packages/trpc/routers/bookmarks.ts b/packages/trpc/routers/bookmarks.ts
index e010569a..6fa6fa91 100644
--- a/packages/trpc/routers/bookmarks.ts
+++ b/packages/trpc/routers/bookmarks.ts
@@ -30,7 +30,10 @@ import {
} from "@karakeep/shared-server";
import { SUPPORTED_BOOKMARK_ASSET_TYPES } from "@karakeep/shared/assetdb";
import serverConfig from "@karakeep/shared/config";
-import { InferenceClientFactory } from "@karakeep/shared/inference";
+import {
+ EmbeddingClientFactory,
+ InferenceClientFactory,
+} from "@karakeep/shared/inference";
import logger from "@karakeep/shared/logger";
import { buildSummaryPrompt } from "@karakeep/shared/prompts.server";
import { EnqueueOptions } from "@karakeep/shared/queueing";
@@ -1031,7 +1034,7 @@ export const bookmarksAppRouter = router({
const hasQueryText = parsedQuery.text.trim().length > 0;
let semanticInfraError: unknown;
const semanticClients = await (async () => ({
- inferenceClient: InferenceClientFactory.build(),
+ embeddingClient: EmbeddingClientFactory.build(),
vectorStoreClient: await getVectorStoreClient(),
}))().catch((error: unknown) => {
if (input.searchMode === "semantic") {
@@ -1040,14 +1043,14 @@ export const bookmarksAppRouter = router({
semanticInfraError = error;
return null;
});
- const inferenceClient = semanticClients?.inferenceClient;
+ const embeddingClient = semanticClients?.embeddingClient;
const vectorStoreClient = semanticClients?.vectorStoreClient;
- if (!hasQueryText || !inferenceClient || !vectorStoreClient) {
+ if (!hasQueryText || !embeddingClient || !vectorStoreClient) {
if (input.searchMode === "semantic") {
throw new TRPCError({
code: "BAD_REQUEST",
message: hasQueryText
- ? "Semantic search requires configured inference and vector store providers"
+ ? "Semantic search requires configured embedding and vector store providers"
: "Semantic search requires a non-empty text query",
});
}
@@ -1074,14 +1077,14 @@ export const bookmarksAppRouter = router({
const semanticSearch = async (limit: number) => {
const embeddingResponse =
- await inferenceClient.generateEmbeddingFromText([
+ await embeddingClient.generateEmbeddingFromText([
parsedQuery.text,
]);
const vector = embeddingResponse.embeddings[0];
if (!vector) {
throw new TRPCError({
code: "INTERNAL_SERVER_ERROR",
- message: "Inference provider returned no query embedding",
+ message: "Embedding provider returned no query embedding",
});
}
return vectorStoreClient.search({