Qwen3.8-Omni-Flash on Voice Chat Server: 1.3GB to 118MB RAM and Instant Wakeup
Contents
On our continuously running voice chat server (Ryzen 7 5800HS, RTX 3050 Ti Laptop 4GB, 16GB RAM), we run STT, LLM, and TTS side by side to respond to voice from an M5Stack CoreS3.
Because Irodori-TTS (voice synthesis) occupies about 3.9GB of the 4GB VRAM, speech recognition (STT) was assigned to Qwen3-ASR-0.6B running on CPU, while the LLM was called via the ModelScope Ambassador API.
However, waking up after being idle took 10 to 16 seconds.
As investigated in our previous article, Windows working-set trimming paged out the idle Qwen3-ASR process, and the first STT call stalled.
Then we spotted an unannounced addition in ModelScope’s OpenAI-compatible inference model list: Qwen-Ambassador/Qwen3.8-Omni-Flash.
If direct audio input works, we can remove the resident CPU ASR model entirely and handle everything from voice input to reply text generation directly in the cloud.
We tested speed, response quality, and memory consumption using test scripts and our server implementation.
flowchart TD
A[CoreS3 Voice Input] --> B{Server Config<br/>pipeline}
subgraph Legacy: stt mode
B -->|stt| C[Qwen3-ASR<br/>CPU resident 1.3GB]
C -->|Text| D[Qwen3.7-Plus<br/>ModelScope API]
end
subgraph New: omni mode
B -->|omni| E[Qwen3.8-Omni-Flash<br/>ModelScope API]
E -.->|Parallel Call| F[Transcript Extraction<br/>Memory & Logging]
end
D --> G[Irodori-TTS<br/>GPU resident]
E -->|Text Streaming| G
G --> H[CoreS3 Voice Output]Test Environment
| Item | Details |
|---|---|
| Voice Chat Server | Windows 11 Home, Ryzen 7 5800HS, RTX 3050 Ti Laptop 4GB, 16GB RAM |
| Python | 3.12 (managed via uv) |
| Model | Qwen-Ambassador/Qwen3.8-Omni-Flash (Ambassador quota) |
| API Endpoint | https://api-inference.modelscope.ai/v1 (OpenAI-compatible) |
| Voice Synthesis (TTS) | Aratako/Irodori-TTS-500M-v3 (GPU resident, port :8355) |
Testing Omni API Standalone
We first tested text input, audio input, and audio output individually using the standard OpenAI Python client.
from openai import OpenAI
client = OpenAI(
base_url="https://api-inference.modelscope.ai/v1",
api_key="...",
)Thinking Behavior
When sending a short text prompt (“Hello. Please introduce yourself in one sentence.”), non-streaming took 5.58 seconds and consumed 214 tokens for a single-sentence response.
Reasoning tokens (thinking) are enabled by default, generating internal thoughts even for trivial greetings.
Setting extra_body={"enable_thinking": False} reduced time to the first streaming chunk.
Because voice chat demands minimal latency, we explicitly disabled thinking for all subsequent calls.
Audio Input Accuracy and Speed
We passed WAV audio (Base64-encoded) using the OpenAI-compatible input_audio format and instructed the model to transcribe it verbatim.
For inputs, we used 48kHz mono WAV files generated by Irodori-TTS (9.88s and 3.68s).
messages = [{
"role": "user",
"content": [
{
"type": "input_audio",
"input_audio": {
"data": "data:audio/wav;base64," + b64_audio,
"format": "wav",
},
},
{"type": "text", "text": "Transcribe the audio verbatim. Output only the transcript."},
],
}]| Input Audio | Ground Truth | thinking | Time to First Char | Total Time | Output Tokens | Transcript Output |
|---|---|---|---|---|---|---|
| Audio 1 (9.88s) | “申し訳ありませんが、現在地がわからないため、天気をお伝えできません。お使いの地域の都市名を教えていただければ、お調べします。“ | ON | 9.94s | 10.30s | 174 | Exact match with ground truth |
| Audio 1 (9.88s) | Same as above | OFF | 3.94s | 4.27s | 25 | Matched text without punctuation |
| Audio 2 (3.68s) | “日本で一番高い山は富士山です。“ | ON | 4.61s | 4.69s | 153 | ”日本で一番高い山は 富士山です。“ |
| Audio 2 (3.68s) | Same as above | OFF | 2.14s | 2.29s | 10 | ”日本で一番高い山は富士山です。” |
Disabling thinking roughly halved the time to the first character.
There were zero transcription errors; for a 3.68s audio clip, output began in 2.14 seconds.
While punctuation was occasionally skipped with thinking turned off, it was fully usable for conversational comprehension and logging.
Audio Output Attempts Trigger 400 Errors
Direct voice generation is a key feature of Omni models, so we tested whether audio output could be received through ModelScope’s inference API.
# Testing audio modality request
client.chat.completions.create(
model="Qwen-Ambassador/Qwen3.8-Omni-Flash",
messages=[{"role": "user", "content": "Hello, greet me in one sentence."}],
stream=True,
modalities=["text", "audio"],
audio={"voice": "Cherry", "format": "wav"},
)We tested six parameter variations:
| # | Parameters | Result |
|---|---|---|
| A | modalities=["text","audio"], audio={...}, streaming | 400 Error (The current model does not support the modalities parameter containing audio.) |
| B | modalities=["audio"], streaming | 400 Error (The model does not support the modalities without text.) |
| C | Same parameters as A, non-streaming | HTTP 200 returned, but choices: null with empty payload |
| D | audio parameter only, no modalities | HTTP 200 returned text only (no audio) |
| E | modalities passed in extra_body | 400 Error (same as A) |
| F | voice="Ethan", format="pcm16" | 400 Error (same as A) |
The error messages suggest that ModelScope’s inference gateway currently restricts audio output parameters rather than a model capability limitation.
For now, we treat Omni as an “audio in -> text out” engine, keeping local Irodori-TTS (GPU resident) for voice synthesis.
Comparing Omni with the Legacy STT+LLM Pipeline
We compared response speed, answer fidelity, and persona behavior between our legacy setup and the Omni pipeline.
Test Setup
| Item | Condition |
|---|---|
| Common Settings | thinking OFF, streaming enabled |
| Input Audio | 6 questions (q0 to q5) synthesized with Irodori-TTS, 2 runs each (12 turns total) |
| Persona Prompt | Character “Kana” (friendly 20s female, max 2 sentences, conversational tone, no emojis or lists) |
| Config A | Local Qwen3-ASR (CPU) transcription -> Qwen3.7-Plus (ModelScope API) |
| Config B | Local Qwen3-ASR (CPU) transcription -> Qwen3.8-Max (ModelScope API) |
| Config C | Qwen-Ambassador/Qwen3.8-Omni-Flash (direct audio input) |
| Measured Metric | Time to complete the first sentence (A and B include local STT processing time) |
Speed Benchmark
The first turn in Run 0 incurred a 10.38-second local STT cold-start delay, so statistics were calculated over the 11 warm turns.
| Configuration | First Sentence (Mean) | First Sentence (Median) | Min – Max | Full Response (Mean) |
|---|---|---|---|---|
| A: STT -> Qwen3.7-Plus | 3.56s | 3.57s | 3.23 – 3.94s | 3.71s |
| B: STT -> Qwen3.8-Max | 3.68s | 3.75s | 3.19 – 4.06s | 3.89s |
| C: Omni-Flash (Direct Audio) | 3.06s | 2.71s | 1.99 – 4.30s | 3.26s |
Note: Standalone local STT took 2.17s on average (range 1.87 – 2.82s).
Under warm conditions, Omni completed the first sentence 0.5s faster on average and 0.9s faster at the median.
The fastest trial delivered sentence one in 1.99s.
However, API response variance was wider than A and B, peaking at 4.30s on one trial.
Because total round-trip time (from mic input to TTS playback completion) takes roughly 12 seconds—with TTS synthesis taking about 7.6 seconds—a 0.5s improvement under warm conditions is barely noticeable in everyday conversation.
The biggest difference appeared during cold starts after idle periods.
While local STT suffered a 10.38s page-fault delay (and up to 16.8s in earlier tests) to reload its working set, Omni sent audio directly to the API and produced the first sentence in 2.5s on the very first turn.
Response Quality and Persona Consistency
Responses across the six questions revealed clear behavioral differences:
| Question | Config A: Qwen3.7-Plus | Config B: Qwen3.8-Max | Config C: Omni-Flash |
|---|---|---|---|
| Mount Fuji height | Correct (3,776m) | Correct (3,776m) | Correct (used kanji numerals on run 2) |
| 345 + 78 | Correct (423) | Correct (423) | Correct (423) |
| Tokyo weather | Declined (no live data) | Declined (no live data) | Declined (no live data) |
| Curry ingredients | Broad (“Meat, veggies, roux”) | Specific (onions, carrots, potatoes, etc.) | Specific (meat, onions, carrots, potatoes, etc.) |
| ”Wake me up at 7 AM tomorrow” | False promise (accepted twice) | Mixed (accepted once, declined once) | Declined (lacks alarm function) |
For the arithmetic question (345 + 78), local STT mistranscribed the Japanese word for plus (“足す”) into the English word “plus”.
While both text LLMs managed to infer the intent, it demonstrated the fragility of cascading STT errors.
Omni interprets acoustic features directly, eliminating cascading transcription errors.
When asked to set an alarm without having system tools, 3.7-Plus falsely agreed both times (“Sure, I will wake you up at 7!”).
Omni refused on both runs, explaining that it had no alarm capabilities and advising the user to set a phone alarm.
Regarding persona consistency, 3.8-Max had the strongest personality but frequently repeated fixed catchphrases like “I will always be right beside you.”
Omni maintained a stable, conversational tone without relying on repetitive catchphrases.
Preserving Transcripts with Parallel Execution
Bypassing local STT creates one architectural problem: the user’s spoken transcript is never saved on the server.
Our voice chat server relies on spoken text for two essential features:
- Long-term memory retrieval using Qdrant (embedding the user’s utterance to query past episodes)
- Conversation history logging
If we instruct Omni to “first output the transcript, then output your reply,” reply generation is delayed by 2 to 3 seconds, stalling TTS playback.
To solve this, voice_chat_server.py runs reply generation and transcription in parallel.
from concurrent.futures import ThreadPoolExecutor
aux_pool = ThreadPoolExecutor(max_workers=4)
def start_reply(wav_bytes: bytes):
if PIPELINE == "omni":
# 1. Run dedicated transcription in a background thread
transcript_future = aux_pool.submit(omni_transcribe, wav_bytes)
# 2. Start reply streaming immediately without waiting
stream = llm.chat.completions.create(
model=OMNI_MODEL,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [audio_part(wav_bytes)]},
],
stream=True,
extra_body={"enable_thinking": False},
)
return stream, transcript_futureWhile the response streams to the client and feeds into local TTS, a background thread calls Omni once more to extract the transcript.
Because transcription completes while Irodori-TTS synthesizes audio, the server captures user text without adding any latency to the reply.
This consumes two API calls per turn, which easily fits within our monthly quota of 10,000 calls.
Production Switch and Memory Savings
Following these benchmarks, we switched our production voice_chat_server.py to the Omni pipeline.
Setting "pipeline": "omni" in config.json activates Omni mode, while changing it back to "pipeline": "stt" provides an instant fallback to local Qwen3-ASR (CPU).
Comparing system metrics before and after the switch on the production laptop:
| Metric | Legacy (Local STT + 3.7-Plus) | Omni (Direct Audio + Parallel Transcript) | Difference |
|---|---|---|---|
| Server Process RSS | 1,327 MB | 118 MB | ~1.2GB Saved (1/11th) |
| Cold-Start Delay After Idle | 10.38s (up to 16.8s) | ~2.5s | 8 to 14s Faster |
/voice_chat End-to-End Time | 9.3 – 12.0s | 8.1 – 8.5s | 1.0 to 3.5s Faster |
/voice_chat_async First Audio Chunk | ~10.5s | 8.4s | ~2.1s Faster |
Removing the resident Qwen3-ASR model reduced the server process RSS from 1,327MB to 118MB.
On a 16GB laptop running Irodori-TTS 24/7, this freed about 1.2GB of physical RAM and eliminated the 10-second idle page-out delays.