Translated from the original Japanese article
Tech6 min read

M5Stack CoreS3 voice chat: multipart WAV POST from PSRAM, 12s sync round trip

M5StackESP32MicrocontrollerElectronicsHardwareExperiment
IkesanContents

In the previous part, mic recording and playback switching worked on their own.
This time the recording goes to the server, and the reply comes back as speech.

The final version will receive the reply asynchronously, in per-sentence chunks.
This time I use the sync variant (everything comes back in one request) to try two things: whether the 48kHz recording goes through STT as-is, and whether my multipart POST format is right.

Setup

The setup is what was settled in the spec check: the CoreS3 hits the VPS relay that stage 1 switched over to, and the relay forwards over Tailscale to the voice-chat server on a 4GB RTX 3050 Ti at home.

The sync API looks like this.

ItemValue
EndpointPOST /voice-relay.php?action=sync
Uploadmultipart/form-data, field name audio, 48kHz/16bit/mono WAV
Responsetranscript, reply, audio_url (reply WAV) and timings, as one JSON
Behaviorblocks until STT→LLM→TTS all finish (about 12s warm, plus relay)

No SD and no filler this time; the build is just stage 3a’s recording code plus the WiFi/HTTP from stages 1/2.

flowchart TD
    A[Tap] --> B[5s recording<br/>48kHz/16bit/mono]
    B --> C[Hand-written RIFF header<br/>WAV in place]
    C --> D[multipart POST<br/>action=sync]
    D --> E[Receive transcript/reply<br/>show on screen]
    E --> F[Download reply WAV<br/>~650KB]
    F --> G[Playback + lip sync]

A 44-byte RIFF header turns the buffer into a WAV

The recording buffer is just a run of PCM samples, so writing a 44-byte RIFF header in front of it makes it a WAV. No filesystem involved.

// 44バイトのRIFFヘッダ直書き(PCM/16bit/モノラル)
static void write_wav_header(uint8_t* p, uint32_t samples, uint32_t rate) {
  uint32_t dataLen = samples * 2;
  memcpy(p, "RIFF", 4);
  put_u32(p + 4, 36 + dataLen);
  memcpy(p + 8, "WAVEfmt ", 8);
  put_u32(p + 16, 16);          // fmtチャンクサイズ
  put_u16(p + 20, 1);           // リニアPCM
  put_u16(p + 22, 1);           // モノラル
  put_u32(p + 24, rate);
  put_u32(p + 28, rate * 2);    // byte rate
  put_u16(p + 32, 2);           // block align
  put_u16(p + 34, 16);          // bit depth
  memcpy(p + 36, "data", 4);
  put_u32(p + 40, dataLen);
}

Building the whole multipart body in PSRAM

HTTPClient’s POST takes the body as a single buffer, so I concatenated the multipart preamble + WAV + trailer in PSRAM. A 5-second recording is about 480KB of WAV, 481KB in total — an easy fit for the 8MB PSRAM.

static const char* BOUND = "M5VoiceChatBoundary";
String prefix = String("--") + BOUND + "\r\n"
                "Content-Disposition: form-data; name=\"audio\"; filename=\"rec.wav\"\r\n"
                "Content-Type: audio/wav\r\n\r\n";
String suffix = String("\r\n--") + BOUND + "--\r\n";

size_t total = prefix.length() + 44 + pcmLen + suffix.length();
uint8_t* body = (uint8_t*)heap_caps_malloc(total, MALLOC_CAP_SPIRAM);
// prefix → RIFFヘッダ → PCM → suffix の順にmemcpy

http.addHeader("Content-Type", String("multipart/form-data; boundary=") + BOUND);
http.setTimeout(60000);  // LLM込み
int code = http.POST(body, total);

With the sync variant, HTTPClient is blocked for the whole POST.
The power button on this device is handled in software, so for the ten-plus seconds until the response arrives, the power button does nothing. That breaks the rule of running the power handling in every wait loop, but only here — the async version (next time) waits by polling and runs the power handling between polls, so I’m accepting it for this build. It’s a test build anyway; nobody is mashing the power button mid-request.

The switch-pop fix went in too

This is where I tried the fix for the mic→speaker switching “pop” confirmed in the previous part. The approach is to play 300ms of silence at volume 0 right after begin(), stepping the volume back up in 10 increments while it plays.

static int16_t s_silence[14400];  // 300ms @48kHz、bssでゼロ初期化

static void speaker_restore() {
  M5.Speaker.begin();
  M5.Speaker.setVolume(0);
  M5.Speaker.playRaw(s_silence, 14400, 48000, false);
  for (int i = 1; i <= 10; i++) {
    M5.Speaker.setVolume(target * i / 10);  // 250msかけて階段状に復帰
    delay(25);
  }
}

A serial command soft 0/1 toggles it; the default is on.

On-device test

First run over serial with recnow (a debug command that starts record→send without touching the screen), using nothing but the ambient noise of the room at night.

録音完了: 240000 samples
sync: POST 480187 bytes (WAV 480044 bytes)
sync: HTTP 200 (12401 ms)
聞: はい。
答: はい、準備できました。何かお手伝いできることはありますか。
返答WAV取得中...
返答WAV: 649004 bytes
返答再生中

(聞: is the transcript, 答: the reply.)

STT accepted the 48kHz/16bit/mono recording with nothing but the header written on it, and the multipart body went through untouched, first try.
In a recording that contained only room noise, STT heard 「はい。」 (“yes”), and the LLM dutifully replied 「はい、準備できました。何かお手伝いできることはありますか。」 (“Yes, I’m ready. Is there anything I can help you with?”). For a moment I had no idea what it had reacted to, which was slightly creepy.

The timings in the response break it down.

SegmentMeasured
STT (Qwen3-ASR)2.2s
LLM (first token)1.8s
LLM (full reply)2.1s
TTS wait5.4s
Server total9.7s
Round trip seen from the CoreS3 (480KB upload + relay)12.4s
Reply WAV (649KB) download~3s

Testing with a real voice

I tapped and spoke the test phrase used since the server build: 「日本で一番高い山は何ですか」 (“What is the highest mountain in Japan?”).

聞: 日本で一番高い山は何ですか。
答: 日本で一番高い山は富士山です。標高は三千七百七十六メートルあります。

The transcript matched the phrase down to the final punctuation mark, so the 48kHz recording gave STT no trouble. The reply WAV (791KB, ~8s) played, and Kana-chan lip-synced through the answer: Mt. Fuji, 3,776 meters.

I tried a question outside the test data too. 「おすすめの朝ごはんを教えてください」 (“What do you recommend for breakfast?”) came back with natto rice and miso soup for nutrition, or banana and yogurt for something quick. A non-canned question still made the round trip in one go.

This round trip took 13.8s (server side STT 2.0s / LLM 2.8s / TTS wait 6.6s), plus about 4s of download.
From tap to the reply starting to play, including the 5s recording: a bit over 20 seconds. The sync version has no filler, so I sat through those 20 seconds in silence.

The question carried over from last time — is 5 seconds even enough — turned out to just barely fit a single spoken question. But with the 20-second wait on top of it, the record-start cue and the duration get tuned in the real build.

The other carry-over, the switching pop, was inaudible in this build with the leading silence + stepped volume restore in place.

Buying a physical button for the record control

With the round trip working, I looked at how to solve the fixed-5-second recording problem: no cue, fixed length.

Auto-detecting the end of speech by silence was rejected first, because on a microcontroller it means re-tuning thresholds for every environment.
A toggle (tap to start, tap again to stop) was also a candidate: the recording length becomes variable, clipped endings disappear, and the current controls (long-press = screen toggle, flick = volume) stay untouched.

But this unit has DIN BASE Ports B and C free. Port A is reserved (the SCD41 CO2 sensor moves back there eventually), which still leaves two GPIO ports idle, so a physical button goes in.

The part is M5Stack’s mechanical Key unit (U144), confirmed in stock at Marutsu, a Japanese parts retailer.

PartRole
Key unit (U144)The talk button. Mechanical key with a built-in RGB LED — press = record, LED = recording, so one part covers the whole cue
ExtIO2 unit (future)I2C GPIO expander. Hang it off a free port and add tact switches for a “preset question button panel”

The record-timing problem that had been on hold since 3a gets solved with a physical key that lights up the moment it’s pressed. If a better idea or some budget shows up, this may still change.
The next stage (async chunks) starts after the button arrives, built around the physical button from the start.

Progress this time

  • Hand-written RIFF header WAV goes through STT
  • multipart POST (field name audio, body built in PSRAM)
  • Sync round trip measured (HTTP 200, 12.4–13.8s, timings retrieved)
  • transcript/reply parsed from the JSON and shown on screen
  • Reply WAV (649–791KB) downloaded and played with lip sync
  • Real-voice accuracy → 「日本で一番高い山は何ですか。」 transcribed down to the punctuation
  • 5s recording → one question just barely fits (cue and duration to be tuned in the real build)
  • Switching pop → inaudible with the fix on (leading silence + stepped volume restore)

Next up

  • action=async: start filler playback + lip sync the moment the job_id arrives
  • Poll action=job every 0.5–1s
  • Download the chunks in order and play them without gaps
  • Restructure so the power button works during POST
  • Move the record control to the Key unit (U144) button + recording LED