I’ve been reading a lot about search lately: kNN, BM25, RRF and related ideas, both within Elasticsearch and in isolation. With the advent of LLMs, especially open-weight models, plus modern hardware, it feels like a good time to approach some of this in a pragmatic way.
On the blog I'm already using a form of BM25 via SQLite’s FTS5 extension , which enables full-text search, similar to the pg_search gem which uses textsearch on Postgres.
What I wanted to do next was to experiment with embeddings and enable a form of semantic search too.
If you want to try it out, it’s live at /semantic-search
While searching around, I came across Alex Garcia's sqlite-vec. He describes it as “a no-dependency SQLite extension written entirely in C that runs everywhere”. That was appealing in itself since I did not want to introduce a separate client/server vector database for this small Rails app. The stable path used here performs brute-force search, which felt perfectly reasonable at my scale. As of June 13, 2026, sqlite-vec’s official ANN tracking issue #25 remains open; however, the v0.1.10 alpha releases introduced experimental DiskANN/ANN work. That experimental support is distinct from the stable brute-force approach in this article.
The next step was to use a model from Hugging Face, create embeddings for all my articles, then enable kNN search using a local Docker container on my server.
Benchmarks
I tested: EmbeddingGemma , Multilingual-E5-large and *E5-base . As a benchmark, I used my own articles and evaluated the results manually with a sample size of one: myself. This was an informal comparison on one corpus and machine, not a controlled or generalizable benchmark.
| Memory ratio comparison (lower is better) | Ratio |
|---|---|
| embeddinggemma-300m / e5-base | 1.16x |
| e5-large / e5-base | 2.07x |
| e5-large / embeddinggemma-300m | 1.78x |
Maximum ratio: 2.07x.
Memory-wise, *E5-large eats the most RSS memory (~4GB), E5-base is the lightest and EmbeddingGemma is right in the middle (~2GB). When it comes to actual results, Gemma returned similar results as *E5-large, at least according to my empirical results tested on my data.
I ran some benchmarks on latency on my MBP M1 Pro. CPU run, cold cache (the model was loaded into memory on each run), one process: model load + one encode:
| Model | Dimensions | Memory | Time |
|---|---|---|---|
| intfloat/multilingual-e5-base | 768d | 1.95 GB | ~6.47s |
| google/embeddinggemma-300m | 768d | 2.26 GB | ~6.24s |
| intfloat/multilingual-e5-large | 1024d | 4.03 GB | ~7.37s |
I settled on Gemma because it was much lighter than *E5-large while giving similarly good results on my own data.
Rails implementation
In order to use this in my Rails blog, I decided the best approach would be to have a separate Docker image for a small service running Gemma with two API endpoints:
+--------------------------------------------------------------------------+
| EMBEDDING API FLOW |
+--------------------------------------------------------------------------+
| GET /health |
| | |
| v |
| 200 OK |
| |
| POST /embed |
| { "query": "text to embed" } |
| | |
| v |
| 200 OK |
| { |
| "embedding": [0.01, -0.02, ...], |
| "dimensions": 768, |
| "model": "google/embeddinggemma-300m" |
| } |
+--------------------------------------------------------------------------+
This runs within the Docker network created by Kamal so it will be accessible by the Rails app. My VM has 6GB of RAM and two Ampere Arm vCPUs; RAM-wise, it should fit with some headroom for the Rails app too.
The current performance on the VM looks like this:
When Article ID Operation Source Status Duration Dims Bytes Model Error
2026-03-29 23:30:03 46 upsert callback success 3368ms 768 8819 google/embeddinggemma-300m -
2026-03-29 23:28:12 46 upsert callback success 3428ms 768 8819 google/embeddinggemma-300m -
...
etc.
| Latency range (ms) | Count |
|---|---|
| 0-999 | 2 |
| 1000-1999 | 18 |
| 2000-2999 | 10 |
| 3000-3999 | 15 |
| 4000-4999 | 4 |
| 5000-6999 | 1 |
The median response time for the API is ~2726 ms, RAM: ~2.5GB in use by both the embedding service and two Puma workers. Given this is a really low-spec VM, I'm quite happy with the experiment so far given there's no cache at all in the hot path.
On the app side, the implementation is fairly simple: add the "sqlite-vec" gem, a route for testing the kNN search /semantic-search and a section in the admin panel where I can trigger the creation of embeddings for my articles.
This is what the Rails flow looks like as an ASCII diagram:
Rails app Embedding service
(SemanticSearch + sqlite-vec) (embedding-inference:8765)
--------------------------- ----------------------------
INDEX / UPSERT FLOW
Admin regen
|
v
ArticleEmbedding#upsert_for(article)
|
+--> available? --no--> trace(skip/error); return false
|
+--> PassageBuilder.build_article(article)
|
+---------------------------> POST /embed {"query":"passage: ..."}
|
v
200 {"embedding":[...],"dimensions":768,"model":"..."}
<----------------------------+
|
+--> validate dims == 768
+--> delete old row + insert new row in article_embeddings
And this is what happens when you click search on /semantic-search :
GET /semantic-search?query=...
|
v
Searcher.search(query)
|
+--> PassageBuilder.build_query(query)
+---------------------------> POST /embed {"query":"query: ..."}
<---------------------------- 200 {"embedding":[...],"dimensions":768}
+--> sqlite-vec KNN MATCH [query_vector], k = limit
+--> join published articles, order by distance ASC
v
Render ranked semantic results
The embedding service
The inference component is documented as the EmbeddingGemma service for Rails and Kamal ; that page summarizes the HTTP contract and deployment boundary. The full source remains available in mp-com-embeddings .
It runs on Python/Torch and uses the sentence_transformers library, small excerpt from the POC:
def load_model(model_name: str, token: str | None, verbose: bool) -> SentenceTransformer:
kwargs: dict[str, Any] = {}
if token:
kwargs["token"] = token
return SentenceTransformer(model_name, local_files_only=True, **kwargs)
This is still an early experiment, but it already feels like a practical complement to FTS5 rather than a replacement for it.
BM25 remains great for exact terms and titles, while embeddings help when the wording drifts but the intent stays the same. Next up are a home-brewed RRF implementation and a more rigorous look at performance.
Notes
- kNN - k-nearest neighbors, a way of finding the closest vectors to a given query embedding.
- BM25 - Best Matching 25, a ranking function commonly used in keyword-based full-text search.
- RRF - Reciprocal Rank Fusion, a simple way to combine multiple ranked result sets, often used in hybrid search setups.