How the database makes embeddings
You can hand Oxid-DB embeddings you calculated elsewhere, or you can send it plain text and let the engine create the embeddings for you. This page is about that second path: what's wired up today, and how to switch it on.
FastEmbed · OpenAI · Ollama: all operationalWhat an embedding is
An embedding turns a piece of text into a list of numbers that captures its meaning. Two things that mean something similar end up with similar numbers, even if they share no words at all. This is what lets the database answer “find me things like this” instead of only “find me this exact word.” It's the machinery behind semantic search and recommendations.
“a dog chasing a ball” and “a puppy playing fetch” land close together. “quarterly tax report” lands far away: no shared keywords required.
Two ways to get them in
There are exactly two paths for getting embeddings into Oxid-DB. Both are fully supported; they just move the work to different places.
You calculate the vectors yourself and hand them over. The database just stores them. Total control, more work on your side, and this always works, regardless of configuration.
You send plain text; the database turns it into an embedding automatically. Less work for you, but first you have to tell it which engine to use. That's the rest of this page.
Three engines + a cache
When you want the database to create embeddings, you pick one of three providers. All three are fully implemented and shipped: no stubs, no experimental gates.
| Engine | What it is | Runs on | Status |
|---|---|---|---|
| FastEmbed | Local ONNX model (BGE small / base) baked into the binary | Your machine | Operational |
| OpenAI | Calls out to OpenAI's cloud embedding API | OpenAI servers | Operational |
| Ollama | Talks to a model you host yourself over HTTP | Your machine / server | Operational |
| CachedEmbedder | An LRU cache that wraps any of the above | n/a | Operational |
FastEmbed is the batteries-included option: no internet, no account, no per-use cost: great for privacy and for getting started. OpenAI trades that for higher quality on many tasks (internet + account + key + per-use billing). Ollama is the middle ground: powerful models, still private and local. CachedEmbedder remembers vectors it has already made, so repeated text doesn't waste time or money.
Is the OpenAI integration fully working?
Yes: it's complete and not stubbed. It handles single and batch requests (including re-ordering OpenAI's out-of-order batch responses), reads the key from an environment variable, times out cleanly, and propagates real errors.
text-embedding-3-small (1536d) · 3-large (3072d) · ada-002 (1536d)OXD_OPENAI_API_KEY; clear error if absentremote feature that is on by default OpenAI and Ollama sit behind a remote Cargo feature, but it's enabled by default and pinned downstream, so both are compiled into the shipped server, CLI, and Python binaries. Nothing turns them off.
The coverage is unit tests, not a live end-to-end call to OpenAI's API (that would need a real key and network). The code path itself is finished: this is a testing gap, not a broken feature.
How auto-embedding gets triggered
Once an engine is configured on the database, there are two ways text turns into a vector automatically:
The main path. You register a mapping, “this text field → embed into this vector collection”, and every document insert embeds it in the background.
Inserting an individual with text: an explicit vector always wins; otherwise, if text is supplied and an engine is configured, that text is embedded.
// on insert, storage checks for a field mapping…if let Some(cfg) = registry.embed_config(collection)&& let Some(embedder) = self.embedder.as_deref(){// …and, if an engine is configured, embeds the textlet vector = embedder.embed_text(&cfg.field_text)?;}
Both mechanisms share one precondition: an embedder must already be configured on the database. No engine → no automatic embedding.
Choosing the engine (and pointing it at your data)
Setup is two steps, done once. First you tell the database which engine to use: the choice is a persisted enum, written through the write-ahead log and rehydrated on load, so it survives restarts. Then you tell it which field to embed.
# 1. Pick an engine: saved to the DB, survives restartsPUT /config/embedder{"provider": "openai","model": "text-embedding-3-small","api_key": "sk-…"}
# 2. Point it at a field: "embed each product's description"oxd collection map products --embed description# from then on, every insert auto-embeds in the backgroundoxd insert products '{ "description": "crisp dry pale lager" }'
PUT /config/embedder · PUT /collections/{name}/mapoxd init --embed · oxd collection map --embedOxidDB(embed=True) · db.set_embed_config(…)The one thing to know: the default
Out of the box, the database is in Manual mode: the “bring your own numbers” mode. There is no default auto-embedder.
In Manual mode, any text you send expecting an automatic embedding is silently ignored: no hash fallback, no low-quality guess, nothing. Pre-computed vectors you supply yourself still work perfectly. “Insert text, get an embedding” only happens after you deliberately configure FastEmbed, OpenAI, or Ollama.
This is a feature, not a surprise: the engine never quietly substitutes a worse embedding behind your back. It does exactly what you configured, and nothing you didn't.
Under the hood: the Embedder trait
Every provider implements one small Rust trait. That's why they're interchangeable: the rest of the database only ever talks to this interface, never to a specific vendor.
pub trait Embedder: Send + Sync {fn dimensions(&self) -> usize;fn embed_text(&self, text: &str)-> Result<Vec<f32>, EmbedError>;fn embed_batch(&self, texts: &[&str])-> Result<Vec<Vec<f32>>, EmbedError> { … }fn name(&self) -> &str;}
Server-side embedding is real and production-wired across FastEmbed, OpenAI, and Ollama. The only catch is that automatic embedding is off until you turn it on and point it at the data you care about.