STT+LLM+TTS voice chat server on RTX 3050 Ti 4GB: 11.9s voice-to-voice

Setup

Here is the hardware and software for this build.

Item	Details
Voice chat server	Ryzen 7 5800HS / RTX 3050 Ti Laptop (4GB VRAM) / 16GB RAM / Windows 11
STT	Qwen/Qwen3-ASR-0.6B (running on CPU)
LLM	Qwen (API access through the Qwen Ambassador program, via ModelScope)
TTS	Irodori-TTS-500M-v3 (resident on GPU)
Server link	VPS and the voice chat server connected over Tailscale

First design: VPS relay

flowchart LR
    A[Device] -->|POST prompt| B[VPS]
    B -->|text generation| C[Qwen]
    C -->|reply text| B
    B -->|request speech synthesis| D[Local TTS server]
    D -->|WAV| B
    B -->|return WAV| A

The VPS and the TTS server at home are connected over Tailscale.

I split the use cases into text-only and text-plus-audio, and made two endpoints on the VPS side.

Moving STT to the same local machine

The design so far assumed the device sends text or audio, but it was missing the STT step that turns microphone recordings into text.
STT models are too big to run on a small device, so it has to happen on a server anyway.
The TTS side was already built, so I put STT on the same machine and let one box handle everything from receiving audio to producing the reply audio.
Compared to relaying text through the VPS, this removes one server from the path.

flowchart TD
    A[Device] -->|POST recorded audio| B[Voice chat server]
    B -->|speech recognition| C[STT: Qwen3-ASR-0.6B]
    C -->|transcript| B
    B -->|text generation| D[Qwen: ModelScope]
    D -->|reply text| B
    B -->|per-sentence TTS| E[TTS: Irodori-TTS]
    E -->|WAV chunks| B
    B -->|return chunks in order| A

STT comparison: kotoba-whisper vs Qwen3-ASR-0.6B

I compared two STT candidates: kotoba-tech/kotoba-whisper-v2.0-faster (faster-whisper, CPU int8) and Qwen/Qwen3-ASR-0.6B (CPU fp32).
TTS already eats almost all of the 4GB VRAM (measured 3,945/4,096 MiB), so STT has to run on CPU.

	kotoba-whisper-v2.0-faster (int8)	Qwen3-ASR-0.6B (fp32)
Model load	31.9s	44.2s
Recognition time (warm)	11.3–14.3s per clip	4.4–8.9s per clip
First clip (incl. warm-up)	21.8s	22.5s
Accuracy	Mostly correct, no punctuation, numbers converted to digits	Exact match, punctuation reproduced

Qwen3-ASR-0.6B was better on speed, accuracy, and punctuation, so I went with it.
The 44-second model load only happens once at server startup.

Per-sentence streaming TTS and turning `enable_thinking` off

I put STT, LLM, and TTS into a single FastAPI server.
There were two things I did to cut the time to the first reply.

First, turning off enable_thinking on the LLM call.
Left on, even a one-line reply comes with 879 characters of thinking and roughly 5 seconds of extra latency (measured: 7.4s down to 2.8s).

Second, splitting the LLM response into sentences and feeding them to TTS as they are generated.
While the LLM is producing sentence two, TTS is already synthesizing sentence one, so the LLM wait and the TTS wait overlap and the total wait shrinks.
The system prompt also tells the model to answer in two sentences or fewer with no symbols, since the output is read aloud.

I measured the end-to-end voice-to-voice time using question audio generated with Irodori-TTS.

	1st run (cold)	2nd run (warm)
STT	20.0s	2.4s
LLM first token	3.0s	1.4s
TTS wait	7.0s	7.6s
Total (voice to voice)	30.3s	11.9s

Both test questions — “What is the highest mountain in Japan?” and “What breakfast do you recommend?” (asked in Japanese) — were recognized with an exact match.

Here is the actual audio. The 1st (cold) run: the question “What is the highest mountain in Japan?” and its reply.

The 2nd (warm) run: the question “What breakfast do you recommend?” and its reply.

Catching a TTS mispronunciation with an STT loopback

I also run the reply audio back through STT to check that it is actually spoken correctly.
In one long sentence, the time expression “10:49” (juu-ji yonjuukyuu-fun in Japanese) was misheard as “19 minutes” — by both STT engines.
Since both engines got it wrong at exactly the same spot, the likely cause is not STT misrecognition but the TTS failing to pronounce “yonjuukyuu-fun” (49 minutes) properly.

Here is one of the clips from the STT bench. You can hear the “10:49” part come out mushy.

Job polling and filler audio to cover the wait

The original synchronous mode returns voice-to-voice in a single request, but chaining STT, LLM, and TTS in series means the response takes several seconds to over ten seconds.
Dead silence on the device side feels wrong, so I rebuilt it as an asynchronous job flow.

The device side is expected to work like this:

Send the recorded audio
Poll the job status every 0.5–1 seconds
Download new chunks as they appear and play them in order

As soon as the job ID comes back, the device starts playing a filler WAV; each poll returns more finished WAV URLs, and a completion flag ends the exchange.

The filler clips — three variations like “Um, let me think for a second.” (in Japanese) — are pre-generated with TTS, and the server exposes their URL list.

Here is the measured timeline from an actual run.

Elapsed	Event
0.0s	Job accepted, job ID returned (device starts filler playback here)
4.7s	Recognized text finalized
10.5s	Sentence 1 WAV done, ready to download and play
14.1s	Sentence 2 done, full reply complete

Sentence 2 is ready before sentence 1 (about 5 seconds of audio) finishes playing, so generation keeps up with playback.
Each chunk is a small self-contained WAV (a few hundred KB for sentence 1), so it fits in a small device’s memory with no streaming decode or buffer management.
The synchronous mode is still there as a fallback.
This whole verification was done by sending pseudo voice recordings from a PC; the device-side firmware is not written yet.

Keeping the character minimal

The system prompt only says the name Kana-chan and one line about a cheerful, upbeat personality.
Last time I over-engineered a character sheet, the replies actually got less natural, so this time I deliberately kept it thin from the start.
The one instruction I did spell out is: no Markdown.
When the output is read aloud, asterisks and heading marks either get spoken literally or linger as text, and both are bad.
In testing, no asterisks or heading marks showed up in the replies.