Qwen3-Embedding Local CPU vs API: +35% TTS Latency, API Adds None
Contents
Last time I measured Qwen3-Embedding-0.6B on CPU and Qdrant’s local mode on my dev machine (i7-13700H, RTX 4060 Laptop 8GB, RAM 32GB).
This time I reran the same measurements on the production machine (AMD Ryzen 7 5800HS, RTX 3050 Ti 4GB, RAM 16GB), and also compared it against calling Qwen3-Embedding-0.6B through the ModelScope ambassador-tier API. I also ran the embedding calls at the same time as Qwen3-ASR (speech recognition) and Irodori-TTS (speech synthesis), both of which stay resident on this machine.
I reconstructed the bench scripts from the methodology described in the previous post (measuring memory via RSS deltas, truncate_dim=256, a 137-character Japanese test string, Qdrant’s local mode via QdrantClient(path=...), and query_points()), since I hadn’t published the code itself.
The SQLite part is a simplified version that just reads back the most recent turns, rather than the source_turn_ids-based restore from the previous post.
Test environment
| Item | Dev machine (previous post) | Production machine (this measurement) |
|---|---|---|
| OS | Windows 11 | Windows 11 Home (10.0.26200) |
| CPU | i7-13700H (14 cores/20 threads) | AMD Ryzen 7 5800HS (8 cores/16 threads) |
| GPU | RTX 4060 Laptop 8GB | RTX 3050 Ti Laptop 4GB (unused this time, CPU only) |
| RAM | 32GB | 15.4GB |
| Python | 3.14.0 | 3.12.10 |
| torch | 2.9.1+cpu | 2.14.0+cpu |
| sentence-transformers | 6.0.1 | 6.0.1 (matches) |
| qdrant-client | 1.19.0 | 1.19.0 (matches) |
Free RAM at the start of the measurement was 6.3GB, close to the ~6.5GB recorded in the previous post.
Two things that tripped me up during setup
Model download failed with OSError: [WinError 1314] A required privilege is not held by the client.
huggingface_hub tries to create a symlink, and Windows without developer mode enabled doesn’t have that permission. Setting HF_HUB_DISABLE_SYMLINKS=1 fixed it (disk usage goes up a bit, but behavior is the same).
qdrant_client.__version__ didn’t exist and raised AttributeError. Switched to importlib.metadata.version("qdrant-client").
Reproducing local CPU execution on the production machine
| Metric | Dev machine (i7-13700H) | Production machine (5800HS) | Verdict |
|---|---|---|---|
| RSS after model load | 529.1 MB | 482.2 MB | close match |
| RSS after first inference | 1,386.5 MB | 1,342.2 MB | close match |
| RSS after 100-item batch | 1,695.0 MB | 1,381.2 MB | somewhat lower |
| Warm, 1 item | 1.114 s | 2.10 s (3-run avg) | ~1.9x slower |
| Warm, 100-item batch (per item) | 1.1789 s | 1.912 s (191.2 s total) | ~1.6x slower |
| Qdrant similarity search | 0.00698 s | 0.00041 s (warm avg) | both under 10 ms |
| Qdrant disk (100 items) | 0.99 MB | 0.41 MB (256-dim) | * |
| SQLite restore | 0.00054 s | 0.00156 s | both under 2 ms |
Model load took 6.56s, first inference 2.10s.
* The smaller disk footprint here is the dimension difference (the API version below uses 1024 dimensions, a different condition).
The previous post’s workload ran to completion on this machine too. The memory footprint (about 1.4GB) matches the previous post closely, but embedding is 1.6-1.9x slower. On a 16GB machine, that’s 1.4GB used by embedding alone at all times.
Measuring the same workload through the ModelScope API
I hit Qwen/Qwen3-Embedding-0.6B at api-inference.modelscope.ai/v1/embeddings. A preflight check also confirmed chat (Qwen-Ambassador/Qwen3.7-Plus) responds in about 2 seconds.
| Metric | Local CPU | Via API | Difference |
|---|---|---|---|
| Single embedding | 2.10 s | 0.83 s (5-run avg, first 0.81 s) | 2.5x faster |
| 100-item batch (per item) | 1.912 s | 0.078 s (one request, 7.8 s total) | ~24x faster |
| Client RSS (end to end) | 1,413 MB | 96.9 MB | ~1/15 |
| Qdrant search (100 items, 1024-dim) | — | 0.00165 s | — |
| Search E2E (embed query + search) | ~2.1 s | 1.20 s | API side faster |
| Qdrant disk (100 items, 1024-dim) | — | 1.0 MB | matches previous post’s 0.99 MB |
The dimensions: 256 parameter also works server-side, and setting it can shrink Qdrant’s storage to a quarter. All 100 items went through in a single request, no splitting needed. encoding_format: "float" is required — omitting it returns 400. Cost, by the Magicube post’s numbers, is 1 coin per embedding call; this whole bench used 9 embedding calls plus 2 chat pings. Against the ambassador tier’s monthly cap of 10,000 calls, that’s about 0.1%.
Concurrent execution under a resident TTS server
This production machine also runs a voice-chat server that bundles STT, LLM, and TTS, and Irodori-TTS speech synthesis runs there too. I’ll call this the VoiceChat server below.
With that VoiceChat server resident (free RAM 6.1GB at the start), I fired synthesis requests at TTS continuously while running embedding calls, testing both “Qwen kept local” and “Qwen routed to the API.”
| Phase | Single embedding | 20-item batch (per item) | TTS synthesis | Free RAM |
|---|---|---|---|---|
| A: TTS alone (baseline) | — | — | 2.5 s (first 4.6 s) | 6.1 GB |
| B: local embedding alone (TTS idle) | 1.84 s | 1.76 s | — | 4.6 GB |
| C: TTS synthesizing + local embedding | 2.64 s (+44%) | 2.22 s (+27%) | 3.4 s (+35% worse) | 4.7 GB |
| D: TTS synthesizing + API embedding | 0.80 s (no change) | 0.075 s | 2.5 s (no degradation) | 5.6 GB |
TTS inference runs on GPU, but its pre/post-processing runs on CPU, so it competes with local embedding for CPU. That’s why TTS synthesis itself also slows by +35% (C), on top of the embedding hit. The API config doesn’t touch the CPU at all — it’s just waiting on network I/O — so its numbers don’t move while TTS is synthesizing (D).
RAM: keeping the local model resident drops free RAM from 6.1 to 4.6GB (about 1.5GB used), recovering to 5.6GB after unload. Add that 1.5GB permanently to a machine that’s already sitting at ~6GB free, and — as the full-pipeline measurement below shows — free RAM drops under 2GB.
The VoiceChat server also has Qwen3-ASR-0.6B (CPU) loaded and resident, but it stayed idle for this measurement (no STT inference running).
At this point, commit memory (the virtual memory the OS has promised to back) was TTS 8.8GB + VoiceChat 5.6GB — large — and the idle portion sat paged out (unused memory swapped to disk).
Concurrent execution with the full ASR→LLM→TTS pipeline
I looped the VoiceChat server’s voice-conversation flow (audio → Qwen3-ASR (CPU) → ModelScope LLM → Irodori-TTS → audio) while running embedding calls.
The input audio was a WAV generated by TTS, looped back through, so this includes ASR’s CPU inference too.
Baseline (voice conversation alone, warm) was 9.3s, broken down as STT 2.3s, LLM first token 1.5s, TTS wait 5.2s. The very first call took 31.6s, with STT alone accounting for 16.8s of that.
| Configuration | Voice E2E | Single embedding | Free RAM |
|---|---|---|---|
| Voice conversation alone (warm) | 9.3 s | — | 3.3 GB |
| + local embedding | 9.5-11.9 s (avg 10.9 s) | 2.31 s (+25%) | 1.73 GB |
| + API embedding | 8.9-9.8 s (no change) | 0.77 s (no change) | 2.66 GB |
Breaking it down: STT ran 2.2-2.9s (avg 2.7s) and TTS wait 5.0-7.0s (avg 6.3s), both heavier than the 2.3s/5.2s baseline. The LLM’s (ModelScope API) first token stayed around 1.5s, unaffected — the CPU contention only shows up in STT and TTS’s pre/post-processing. With the API config, that whole breakdown didn’t move either, since it’s purely network I/O.
Free RAM with the local config dropped to 1.73GB. With ASR actually paged back in, the 1.5GB embedding model on top of it pushed things close to the edge.
The first call to ASR after it had been paged out took 31.6s.
The cause was confirmed to be Windows working-set trimming (the OS shrinking a process’s memory footprint while it’s idle): after sitting idle overnight, the VoiceChat process’s working set had shrunk to 10MB against a 5.6GB commit, and ballooned to 2,614MB right after hitting STT (meaning 16.8s of that first call was just reading pages back in from the page file).
This isn’t a model or server bug — it’s that the total committed memory across resident processes on this machine is too much for 16GB. Re-measured during the full-pipeline run, it came out to TTS 10.3GB + VoiceChat 5.1GB (a separate reading from the TTS-alone measurement in the previous section).
The fix looks like either pinging STT with a dummy call every few minutes (keep-warm) or cutting the total commit memory footprint itself — routing embedding to the API would drop the resident model’s ~1.5GB right there, and quantizing ASR to fp16/int8 is another candidate.
What to run on the production machine
The four items I’d left unconfirmed in the previous post are now answered.
| Question from the previous post | Result |
|---|---|
| Has free RAM changed from the ~6.5GB on record? | 6.3GB, close match |
| Actual RAM increase and processing time | ~1.4GB, 1.6-1.9x slower per item (vs. dev machine) |
| Qdrant local mode insert/search speed | both under a few milliseconds |
| Does running alongside STT/TTS cause OOM or slowdown? | No OOM, but voice replies get +1.5-2.5s slower per turn and free RAM drops to 1.73GB |
For the voice pipeline’s long-term memory, embedding goes to the ambassador-tier API (with dimensions: 256), and only Qdrant’s local mode (RSS overhead in the tens of MB, search under 2ms) stays on the machine.
Local CPU execution is worth keeping as an offline fallback, but there’s no reason to run it resident.
The combination of embedding and Qdrant looks workable, between last time and this time.
What’s next is either wiring this long-term memory into where CoreS3 already calls the voice-chat server asynchronously, or building CoreS3 into a servo-driven StackChan Body first. The CoreS3 I have already handles voice conversation and lip sync — it just isn’t in a StackChan Body yet.