Qwen3-Embedding + Qdrant on CPU: 1.1s/query, 1.5-1.7GB RAM
Contents
Update (2026-09-09): Reran the same measurements on the production machine (Ryzen 7 5800HS), and also compared it against the ModelScope API and measured the interference when running alongside a resident TTS/ASR server → Qwen3-Embedding Local CPU vs API: +35% TTS Latency, API Adds None
Last time I locked down the design for long-term StackChan memory with Qwen and Qdrant.
The RTX 3050 Ti laptop running the voice-chat server has VRAM at 3,945/4,096 MiB, so there’s no room left to add Qwen3-Embedding on the GPU.
I’d already decided to run the 0.6B version on CPU, but hadn’t yet measured “RAM usage and processing time, before worrying about storage size.”
So instead of touching that production machine, I measured the two new costs first on my dev machine (Windows 11, RTX 4060 Laptop 8GB, RAM 32GB): Qwen3-Embedding-0.6B on CPU, and Qdrant’s local mode.
This dev machine and the production machine are different on both CPU and GPU, so these numbers don’t transfer directly.
RAM usage should depend mostly on model size and be fairly machine-independent, so that’s the part I focused on. Processing time depends heavily on this machine’s CPU, so I’m treating it as a reference value only.
Test environment
Here’s the setup.
| Item | This dev machine | Production machine (reference) |
|---|---|---|
| OS | Windows 11 | Windows 11 |
| CPU | 13th Gen Intel Core i7-13700H (14 cores/20 threads) | AMD Ryzen 7 5800HS (8 cores) |
| GPU | NVIDIA GeForce RTX 4060 Laptop 8GB | RTX 3050 Ti Laptop (VRAM 4096MiB, 3945MiB in constant use) |
| RAM | 32GB | 16GB |
| Python | 3.14.0 | Not stated in earlier posts |
Production machine numbers come from Building an STT+LLM+TTS server on an RTX 3050 Ti 4GB and Trying Irodori-TTS Japanese voice cloning on an RTX 3050 Ti Laptop with 4GB VRAM.
Setup
I cut a working venv with python -m venv --system-site-packages .venv and reused the globally installed torch, numpy, and psutil. torch was already the CPU build.
Then I added sentence-transformers and qdrant-client. Versions:
| Package | Version |
|---|---|
| torch | 2.9.1+cpu |
| numpy | 2.3.3 |
| psutil | 7.1.3 |
| sentence-transformers | 6.0.1 |
| transformers | 5.16.1 |
| qdrant-client | 1.19.0 |
| huggingface-hub | 1.30.0 |
| scikit-learn | 1.9.0 |
| pydantic-core | 2.46.5 |
| tokenizers | 0.23.2 |
That’s the state I measured from.
Running Qwen3-Embedding-0.6B on CPU
The test text was about the same length as the JSON example from the previous post: 137 characters of Japanese combining topic, summary, and entities. With this, I measured RSS (the process’s actual resident memory usage) before and after model load, and embedding time for 1, 10, and 100 items.
RSS climbed more after the first real inference than it did after just loading the model.
| Point | RSS |
|---|---|
| Before import | 19.2 MB |
| After importing sentence_transformers | 404.9 MB |
| After model load | 529.1 MB |
| After first inference | 1386.5 MB |
| After 100-item batch | 1695.0 MB |
Processing time is only long for the load that includes the first download, which is mostly network wait. Once warmed up, single items and batches took about the same time.
| Item | Time |
|---|---|
| Model load (including first download) | 193.50s |
| First inference (including lazy init) | 1.22s |
| Warm, 1 item | 1.114s |
| Warm, 10-item batch (per item) | 1.0753s |
| Warm, 100-item batch (per item) | 1.1789s |
Warm, 1 item (truncate_dim=256) | 1.1208s |
With sentence-transformers’ default settings, batching barely shortened the per-item time on CPU — 1.08 to 1.18 seconds/item across the board. Dropping the dimension from 1024 to 256 with Matryoshka Representation Learning (MRL) didn’t change this either. MRL only truncates the output vector; the model itself does the same amount of work.
The benefit of dropping the dimension is a smaller stored vector (4096 bytes down to 1024 bytes) and less memory/disk on the Qdrant side. It doesn’t make embedding itself faster.
I first tried passing truncate_dim=256 as a constructor argument to SentenceTransformer. That built a separate instance alongside the existing 1024-dim model, so RSS from both instances added up. Switching to passing it at call time with model.encode(texts, truncate_dim=256) kept it to a single instance.
Running Qdrant in local mode
QdrantClient(path=...) gives a local mode that persists to disk without running a server. I generated dummy data matching the searchable-memory fields from the previous post, embedded it with Qwen3-Embedding, and inserted 100 records.
My first script used QdrantClient.search(), but qdrant-client 1.19.0 has removed search(), so it raised AttributeError. I switched to query_points() and pulled results from resp.points instead, then reran it.
| Item | Value |
|---|---|
| Vectorizing 100 records | 0.5997s/record |
| Upsert of 100 records | 0.429s |
| Similarity search only (top 5) | 0.00698s |
Search with entities filter (top 5) | 0.00370s |
Disk usage of qdrant_data directory | 0.99 MB |
| RSS with both running (after upsert) | 1562.1 MB |
The filtered search on entities was faster than the unfiltered one — at 100 records, the reduced search space seems to outweigh the filtering overhead. Combined RSS for the embedding model and Qdrant together comes out to roughly 1.5-1.7GB on this dev machine.
Searching for “what did I run on the RTX 3050 Ti laptop” pulled the only dummy record mentioning the RTX 3050 Ti — the “StackChan voice backend” topic — into the top 3 results.
| Rank | Score |
|---|---|
| 1st | 0.4762 |
| 2nd | 0.4692 |
| 3rd | 0.4667 |
Getting the original turn back out of SQLite
Up to this point I’d only measured embedding and Qdrant on their own — I hadn’t tried the design from the previous post, where source turns stay in SQLite and Qdrant holds only the summary vectors and search fields. So I built the SQLite schema for real and checked whether a search result could actually get back to the original turn in SQLite.
Two tables: turns and memories.
| Table | Columns |
|---|---|
turns | turn_id, speaker, text, occurred_at |
memories | memory_id, topic, summary, entities, memory_type, occurred_at, source_turn_ids, status, qdrant_point_id |
memories.qdrant_point_id links to the matching point on the Qdrant side.
Five topics went in: the voice backend, buying a PC, wiring up the CO2 monitor’s sensor, a favorite drink, and a chat about Qdrant filters — each repeated 4 times for 20 topic instances. Each topic carries one user turn and one assistant turn, so that’s 40 turns and 20 memories. Summaries came from a fixed template rather than Qwen, standing in for the real thing.
| Item | Value |
|---|---|
| SQLite write (40 turns + 20 memories) | 0.0030s |
| Vectorizing 20 records | 13.560s (0.678s/record) |
| Qdrant upsert (20 records) | 0.0954s |
| Search (including embedding the query) | 0.2673s |
Restoring source turns from SQLite (via source_turn_ids) | 0.00054s |
| SQLite file size | 24.0 KB |
| Qdrant disk usage | 0.21 MB |
Searching “what did I end up doing about the Fujitsu PC thing” hit the “buying a PC” topic at the top (score 0.6801). Its point ID led through the memories table to the turns table, pulling back the two original turns.
turn12 [user] PCどれ買った?
turn13 [assistant] 富士通のesprimoにした。安定重視で選んだ。The flow from the previous post’s design worked as intended: search the summaries in Qdrant, follow memory_id back to the original turn in SQLite. Restoring from SQLite was sub-millisecond and clearly not a bottleneck — most of the time went to embedding the query text (0.267s).
Does this fit on the production machine?
Matching this dev machine’s measurements (roughly 1.5-1.7GB combined for the embedding model and Qdrant) against the production machine’s free RAM: my notes from when I was testing STT and TTS there put the free RAM at about 6.5GB out of 16GB. Whatever this dev machine uses would fit inside that.
On the time side, the design only calls the embedding once per turn when a search is actually needed, so it’s not a cost that hits every turn. On the turns where it does run, it still adds about a second of latency just for the embedding. The production CPU is a Ryzen 7 5800HS, a different generation from this dev machine’s i7-13700H.
Taking this code to the production machine (the laptop running Irodori-TTS), here’s what I’ll check:
- Whether the current free RAM still matches the ~6.5GB on record
- Actual RAM increase and processing time running Qwen3-Embedding-0.6B on the Ryzen 7 5800HS
- Qdrant local mode insert/search speed
- Whether running it alongside STT/TTS causes OOM or a slowdown