Embedding Generation

AutoMem generates vectors for stored memories in a background worker. The embedding provider is a server setting, independent of where the AutoMem service is deployed. See Choose your setup for the choices to make before storing your first memory.

Key features:

Multi-provider support: Voyage AI, OpenAI, FastEmbed (local), Ollama, or deterministic placeholder vectors
Auto-selection at initialization: Voyage → OpenAI → configured Ollama → FastEmbed → Placeholder
Batch processing: Up to 20 memories per API call with 2-second timeout
Dimension flexibility: Supports common sizes such as 256d, 384d, 512d, 768d, 1024d, 2048d, 3072d — plus Ollama (variable by model) and Placeholder (any configurable size)
Graceful degradation: Falls back to placeholder vectors if all providers fail

For information about how memories are enriched with entities and relationships after storage, see Enrichment Pipeline. For the overall background processing architecture, see Background Processing.

System Architecture

The embedding generation system operates independently from the main API request flow, using a queue-based worker thread to batch-process memories and generate embeddings through a pluggable provider abstraction layer.

When a memory is stored via POST /memory, the endpoint immediately writes to FalkorDB and returns a response without waiting for embedding generation. The embedding job is queued for asynchronous processing.

Provider Selection System

Set EMBEDDING_PROVIDER to choose a provider explicitly. The default, auto, tries the following providers at initialization based on the server environment.

Auto-Selection Priority Chain

flowchart TD
    Request["Initialize embedding provider"]

    Request --> CheckConfig{"EMBEDDING_PROVIDER<br/>config?"}

    CheckConfig -->|"auto (default)"| Priority["Auto-selection priority"]
    CheckConfig -->|"voyage"| ForceVoyage["Force Voyage<br/>Require VOYAGE_API_KEY"]
    CheckConfig -->|"openai"| ForceOpenAI["Force OpenAI<br/>Require OPENAI_API_KEY"]
    CheckConfig -->|"local"| ForceLocal["Force FastEmbed<br/>Local ONNX model"]
    CheckConfig -->|"ollama"| ForceOllama["Force Ollama<br/>Local server"]
    CheckConfig -->|"placeholder"| ForcePlaceholder["Force Placeholder<br/>Hash-based"]

    Priority --> Try1{"VOYAGE_API_KEY<br/>set?"}
    Try1 -->|Yes| Voyage["VoyageEmbeddingProvider<br/>voyage-4: 1024d"]
    Try1 -->|No| Try2{"OPENAI_API_KEY<br/>set?"}

    Try2 -->|Yes| OpenAI["OpenAIEmbeddingProvider<br/>text-embedding-3-small: 1536d native<br/>truncated to VECTOR_SIZE via Matryoshka"]
    Try2 -->|No| Try3{"OLLAMA_BASE_URL<br/>or OLLAMA_MODEL<br/>set?"}

    Try3 -->|Yes| Ollama["OllamaEmbeddingProvider<br/>Local server"]
    Try3 -->|No| Try4{"FastEmbed<br/>available?"}

    Try4 -->|Yes| FastEmbed["FastEmbedProvider<br/>BAAI/bge-large-en-v1.5 at 1024d"]
    Try4 -->|No| Placeholder["PlaceholderEmbeddingProvider<br/>No semantic meaning"]

    Voyage --> Validate["Validate dimension<br/>matches VECTOR_SIZE"]
    OpenAI --> Validate
    Ollama --> Validate
    FastEmbed --> Validate
    Placeholder --> Validate

    ForceVoyage --> Validate
    ForceOpenAI --> Validate
    ForceLocal --> Validate
    ForceOllama --> Validate
    ForcePlaceholder --> Validate

    Validate --> Store["Store in Qdrant<br/>or log warning"]

Provider Capabilities Matrix

Provider	Dimensions	API Key	Where inference runs
Voyage	256, 512, 1024, 2048	`VOYAGE_API_KEY`	Hosted API
OpenAI	Depends on the model and endpoint	`OPENAI_API_KEY`	Hosted or OpenAI-compatible API
FastEmbed	384, 768, 1024	None	Inside the AutoMem service
Ollama	Depends on the model	None	On the configured Ollama server
Placeholder	Any configured size	None	Inside AutoMem, using deterministic hashes

FastEmbed and self-hosted Ollama have no embedding API fees. Plan for the hardware running the model and, for FastEmbed, the initial model download. Hosted embedding APIs receive the memory and query text they embed. Check the provider’s terms and usage rates.

In auto, Ollama is considered when OLLAMA_BASE_URL or OLLAMA_MODEL is set. The server does not check Ollama reachability when selecting the provider. Connection errors surface when it generates embeddings. When earlier providers are unavailable, FastEmbed uses BGE Large at the default 1024 dimensions for new collections. If it cannot load, auto uses deterministic placeholder vectors, which carry no semantic meaning.

Dimension Validation

Dimension reconciliation happens at startup — not on every write:

Default (VECTOR_SIZE_AUTODETECT=true): get_effective_vector_size() in automem/utils/validation.py reads the existing collection’s dimension. If it differs from VECTOR_SIZE, AutoMem logs a warning, adopts the collection’s dimension, and builds the embedding provider at that size.
Strict mode (VECTOR_SIZE_AUTODETECT=false): a mismatch makes get_effective_vector_size() raise VectorDimensionMismatchError (a RuntimeError, not a ValueError) with both dimensions and fix options in the message. In the 0.16.2 startup path, ensure_qdrant_collection() catches that error with its generic exception handler, logs the traceback, and sets the Qdrant client to None, so startup continues without a Qdrant client instead of exiting.
Provider check: once the effective dimension is set, Voyage is the only provider with a hard dimension check at init (_PROVIDER_DIMENSION_CONSTRAINTS): the dimension must be 256, 512, 1024, or 2048. With EMBEDDING_PROVIDER=voyage, any other dimension (for example an adopted 1536d or 3072d collection) raises a RuntimeError and startup fails. In auto mode the same failure is logged as a warning and selection moves on to the next provider.
validate_vector_dimensions() is a legacy wrapper kept for backwards compatibility. app.py imports it but nothing calls it.
Length checks happen where vectors are produced, not at the Qdrant call. generate_real_embeddings_batch() compares every vector with the effective dimension, retries a failed or mismatched batch item by item, and replaces items that still fail with placeholder vectors (or raises when called with allow_placeholder_fallback=False). A client-supplied embedding with the wrong length is rejected with a 400 by coerce_embedding() on POST /memory.
Together these checks prevent Qdrant collection corruption from mixed dimensions when switching providers.
FalkorDB writes always succeed regardless of embedding status — the graph never depends on a healthy vector path.

Provider-Specific Features

Voyage AI Provider

Configuration:

VOYAGE_API_KEY — Required
Model defaults to voyage-4; also supports voyage-4-large and voyage-4-lite. Configurable via VOYAGE_MODEL env var.

Voyage-specific features:

Shared embedding space across voyage-4 family models
MoE architecture in voyage-4-large for best quality
Optimized voyage-4-lite for latency/cost tradeoff
Support for input_type hint (“query” or “document”)
Exponential backoff retry for 429/5xx errors
Maximum batch size: 128 texts per API call

OpenAI Provider

Configuration:

OPENAI_API_KEY — Required
EMBEDDING_MODEL — Default: text-embedding-3-small (also: text-embedding-3-large)
OPENAI_BASE_URL — Optional: Custom endpoint for OpenAI-compatible providers

OpenAI-compatible providers:

Native OpenAI API
OpenRouter (https://openrouter.ai/api/v1)
LiteLLM (http://localhost:4000/v1)
vLLM (http://localhost:8000/v1)

Key implementation details:

dimensions parameter sent only to native OpenAI endpoints
OpenAI-compatible providers do not support the dimensions parameter
Detection via _is_openai_native() helper function
Maximum batch size: 2048 texts per API call (use 20-32 to avoid timeout)

FastEmbed Provider (Local)

Configuration:

EMBEDDING_PROVIDER=local (explicit) or auto-detected when no API keys present
Models cached in ~/.config/automem/models/ (override with AUTOMEM_MODELS_DIR env var)

Model selection by dimension:

Dimension	Auto-Selected Model	Size
384	`BAAI/bge-small-en-v1.5`	~67MB
768	`BAAI/bge-base-en-v1.5`	~210MB
1024	`BAAI/bge-large-en-v1.5`	~1.2GB

Features:

ONNX runtime for fast local inference
No API calls, no network required after initial model download
Warmup batch on initialization
Automatic dimension detection and validation

Placeholder Provider (Deterministic Fallback)

Configuration:

EMBEDDING_PROVIDER=placeholder (explicit) or auto-selected when no providers available

Implementation:

SHA-256 hash of content as RNG seed
Seeded random number generator for reproducibility
Normalized to [0, 1] range
Same text always produces identical embedding
No semantic meaning, but consistent for deduplication

Use cases:

Development without API keys
Testing and CI pipelines
Graceful degradation when all providers fail
Temporary fallback during API outages

Queue-Based Processing

Memory Storage Flow

When a memory is stored via POST /memory:

Flask route handler writes memory to FalkorDB (synchronous, blocks response)
Job is added to embedding_queue as a (memory_id, content) tuple
Response is returned immediately (non-blocking)
Background worker picks up the job and generates the embedding

Job Structure

Each queued job is a two-item tuple:

Position	Field	Type	Description
`0`	`memory_id`	str	UUID of the memory to embed
`1`	`content`	str	Text content to generate embedding for

Batching Strategy

Accumulation Logic

The embedding_worker() function implements a time-boxed accumulation strategy to balance latency and cost efficiency.

The worker accumulates jobs into a batch, then triggers processing when either condition is met:

Batch size reaches EMBEDDING_BATCH_SIZE (default: 20 items)
A queue read times out while a batch is pending. The wait is bounded by a rolling EMBEDDING_BATCH_TIMEOUT_SECONDS deadline (default: 2.0 seconds) that restarts after every flush and every empty-queue timeout, so it is not measured from the moment the first item was added. Each queue read waits at least 0.1 seconds.

This ensures low-traffic periods don’t cause indefinite delays while high-traffic periods maximize API efficiency.

Configuration Parameters

Variable	Default	Description
`EMBEDDING_BATCH_SIZE`	20	Maximum items per batch before forcing processing
`EMBEDDING_BATCH_TIMEOUT_SECONDS`	2.0	Maximum wait time for batch accumulation (seconds)

Batch size limits by provider:

Provider	Maximum Batch Size	Recommendation
Voyage	128	Use 20-50 for cost/latency balance
OpenAI	2048	Use 20-32 to avoid timeout
FastEmbed	Unlimited	Use 20-50 for memory efficiency
Ollama	1 (sequential)	Batch size ignored
Placeholder	Unlimited	Use 20 for consistency

Batch Processing Pipeline

End-to-End Flow

graph LR
    Batch["batch: List[Tuple[str, str]]<br/>(up to 20 jobs)"]
    Extract["Extract content<br/>text: List[str]"]
    Generate["process_embedding_batch()<br/>Provider bulk request"]
    Embeddings["embeddings: List[List[float]]<br/>(dimensions per provider)"]
    Store["store_embedding_in_qdrant()<br/>For each (memory_id, content, embedding)"]
    Success["Log success<br/>+ stats update"]

    Batch --> Extract
    Extract --> Generate
    Generate --> Embeddings
    Embeddings --> Store
    Store --> Success

Bulk Embedding Generation

The process_embedding_batch() function sends multiple texts to the configured embedding provider:

Input: List[str] — Text content from each memory
Output: List[List[float]] — Embedding vectors (dimension depends on provider/model)
Provider: state.embedding_provider.generate_embeddings_batch(texts)

Provider-specific batch processing:

Provider	Batch Method	Internal Batching
Voyage	`client.post()` with `input: List[str]`	Single API call for up to 128 texts
OpenAI	`client.embeddings.create()` with `input: List[str]`	Single API call for all texts
FastEmbed	`model.embed()` iterator	Local ONNX batch inference
Ollama	Sequential `POST /api/embeddings`	One request per text
Placeholder	`[_hash_based_vector(t) for t in texts]`	Local computation, no batching needed

Key implementation details:

Single provider method call for all texts in batch (vs. N separate calls)
Returns embeddings in same order as input texts
Handles provider-specific errors with logging
Falls back to placeholder vectors on provider failure
Validates dimension consistency before storage

Storage Phase

The store_embedding_in_qdrant() helper function persists each embedding.

Payload Requirements:

Must include all searchable fields: content, tags, tag_prefixes, type, importance, timestamp, metadata
Payload fields are fetched from FalkorDB before the Qdrant upsert
Ensures Qdrant can be used as backup/recovery source

Error Handling and Resilience

Retry Logic

Scenario	Behavior
OpenAI API failure	Log error, skip embedding, continue with next batch
Qdrant connection failure	Log warning, memory remains in FalkorDB (graceful degradation)
Batch generation exception	Log error, mark queued jobs done, and rely on the canonical FalkorDB record
Per-memory Qdrant upsert exception	Log error for that memory and continue processing the batch
Queue full	Use `queue.put()` without timeout (blocks until space available)

Graceful Degradation

The system operates in multiple modes based on provider and storage availability:

Optimal: Voyage/OpenAI + Qdrant = Semantic vector search with high-quality embeddings
Good: FastEmbed/Ollama + Qdrant = Semantic vector search with local embeddings
Acceptable: Voyage/OpenAI without Qdrant = Embeddings stored in FalkorDB, keyword search only
Degraded: Placeholder + Qdrant = Deterministic vectors for deduplication, no semantic meaning
Minimal: Placeholder without Qdrant = Graph-only keyword and relationship search

Performance Characteristics

Latency Impact

Ordinary stores queue embedding work after writing the memory record to FalkorDB. Vectors become searchable after the worker processes the queue. Recall queries that use semantic search may need an embedding during the request, so their latency depends on the provider and network.

Cost Reduction

Batching reduces the number of provider requests for queued memories. Hosted providers may bill based on tokens or another usage unit, so fewer requests do not imply a proportional change in cost.

Configuration Reference

Core Embedding Configuration

Variable	Type	Default	Description
`EMBEDDING_PROVIDER`	str	`auto`	Provider selection: `auto`, `voyage`, `openai`, `local`, `ollama`, `placeholder`
`EMBEDDING_BATCH_SIZE`	int	20	Maximum memories per batch
`EMBEDDING_BATCH_TIMEOUT_SECONDS`	float	2.0	Maximum batch accumulation time (seconds)
`VECTOR_SIZE`	int	1024	Embedding dimension used to create a new Qdrant collection. With `VECTOR_SIZE_AUTODETECT=true` (the default), an existing collection’s dimension takes precedence. `text-embedding-3-small` natively produces 1536d but the system passes the effective dimension as `dimensions` to OpenAI’s own API to truncate via Matryoshka representation learning.
`VECTOR_SIZE_AUTODETECT`	bool	`true`	When an existing Qdrant collection’s dimension differs from `VECTOR_SIZE`, adopt the collection’s dimension and log a warning. Set to `false` for strict matching, which raises `VectorDimensionMismatchError` instead (see Dimension Validation).

Provider-Specific Configuration

Variable	Type	Required For	Description
`VOYAGE_API_KEY`	str	Voyage	Voyage AI API key
`VOYAGE_MODEL`	str	Voyage	Voyage model name (default: `voyage-4`; also: `voyage-4-large`, `voyage-4-lite`)
`OPENAI_API_KEY`	str	OpenAI	OpenAI or compatible provider API key
`OPENAI_BASE_URL`	str	OpenAI	Custom endpoint (OpenRouter, LiteLLM, vLLM)
`EMBEDDING_MODEL`	str	OpenAI	Model: `text-embedding-3-small`, `text-embedding-3-large`
`OLLAMA_BASE_URL`	str	Ollama	Ollama server URL (default: `http://localhost:11434`)
`OLLAMA_MODEL`	str	Ollama	Ollama embedding model (default: `nomic-embed-text`)
`OLLAMA_TIMEOUT`	int	Ollama	Request timeout in seconds (default: `30`)
`OLLAMA_MAX_RETRIES`	int	Ollama	Maximum retry attempts on failure (default: `2`)
`AUTOMEM_MODELS_DIR`	str	FastEmbed	Local model cache directory (default: `~/.config/automem/models/`)

Storage Configuration

Variable	Type	Default	Description
`QDRANT_URL`	str	—	Qdrant endpoint (optional)
`QDRANT_API_KEY`	str	—	Qdrant authentication (optional)
`QDRANT_COLLECTION`	str	`memories`	Qdrant collection name

Configuration validation:

VECTOR_SIZE should match the Qdrant collection dimension. By default a differing collection dimension is adopted; with VECTOR_SIZE_AUTODETECT=false a mismatch raises VectorDimensionMismatchError
Provider-specific keys are only required when using that provider
Auto mode needs no API key: without Voyage or OpenAI keys it tries Ollama (when OLLAMA_BASE_URL or OLLAMA_MODEL is set), then FastEmbed, and falls back to placeholder only if FastEmbed fails to load
The dimension must be valid for the selected provider

Tuning Recommendations

High-traffic scenarios (>100 memories/hour):

EMBEDDING_BATCH_SIZE=50, EMBEDDING_BATCH_TIMEOUT_SECONDS=5.0
Maximizes batching efficiency, acceptable latency for bulk operations

Low-traffic scenarios (<10 memories/hour):

EMBEDDING_BATCH_SIZE=5, EMBEDDING_BATCH_TIMEOUT_SECONDS=0.5
Reduces embedding delay, maintains some batching benefit

Real-time requirements:

EMBEDDING_PROVIDER=local, EMBEDDING_BATCH_SIZE=1
Minimizes latency with local inference, sacrifices batching efficiency

Cost-optimized (free tier):

EMBEDDING_PROVIDER=voyage, EMBEDDING_BATCH_SIZE=50
Voyage generous free tier + lite model for best cost/performance

Development/testing:

EMBEDDING_PROVIDER=placeholder
No API keys required, deterministic results for testing

Thread Safety

The embedding worker runs in a dedicated thread started during application initialization:

Daemon thread: Automatically terminates when main process exits
Single worker: One thread handles all embedding jobs
Thread-safe queue: Python queue.Queue provides synchronization
No shared state: Worker only accesses queue and external services

Integration Points

Memory Storage Integration

The POST /memory endpoint integrates with the embedding queue via automem/api/memory.py.

Enrichment Pipeline Coordination

Embedding generation and enrichment run independently:

System	Trigger	Dependency
Embedding Worker	Memory created	None (immediate queue)
Enrichment Worker	Memory created	Uses Qdrant for similarity search (uses existing embeddings)

The enrichment pipeline may query Qdrant for similar memories, so embedding generation should ideally complete before semantic neighbor relationships are created. In practice, they run concurrently and the enrichment worker uses whatever embeddings are already available.

Monitoring and Debugging

Log Messages

The embedding worker logs structured events:

DEBUG   | Generated and stored embedding for <memory_id>
INFO    | Stored embedding for <memory_id> in Qdrant
ERROR   | Failed to generate batch embeddings
ERROR   | Failed to store embedding for <memory_id>
WARNING | Memory <memory_id> not found in FalkorDB, skipping Qdrant update

Recovery and Reprocessing

Manual Re-embedding

The /admin/reembed endpoint allows batch re-embedding of existing memories:

Use cases:

Migrating to different embedding model
Recovering from Qdrant data loss
Fixing corrupted embeddings
Changing from one provider to another

Choose an explicit provider and model before your first memory write. If you change the provider or model later, plan to re-embed the existing corpus when the embedding space changes. Matching vector lengths alone do not make two models compatible.

If the output dimension changes, create a collection at the new dimension or rebuild the existing one. The new provider’s output must match the effective collection dimension. See the re-embedding API for the rebuild operation and Backup & Recovery before migrating a collection.

Starting in AutoMem 0.16.2, new Qdrant collections use on-disk vectors, indexes, and payloads. Existing collections retain their layout until migrated separately.