Translated from the original Japanese article
Tech6 min read

M5Stack CoreS3 voice chat: playWav WAV rules, mic/speaker I2S swap, lip sync

M5StackESP32MicrocontrollerElectronicsHardware
IkesanContents

In the previous post I put together a voice chat server on an RTX3050Ti 4GB. This time the experiment is calling that API from an M5Stack CoreS3 and having the device itself do the talking.
Before that, as prep, this post pins down the specs and implementation steps needed to make it run.

What I’m building

The device side should move through this flow.

flowchart TD
    A[Touch to start recording] --> B[Record from mic]
    B --> C[POST WAV to server]
    C --> D[Receive job ID<br/>play filler audio]
    D --> E[Poll every 0.5-1s]
    E --> F[Download finished WAV<br/>chunks in order]
    F --> G[Play chunks<br/>lip sync while playing]
    G --> H[Done flag ends the turn]

Here is what needs checking.

  • Can it play filler WAVs from the SD card on the built-in speaker?
  • Can it record from the mic, wrap it as WAV, and send it to the server?
  • Can it switch mouth images fast enough for lip sync?

Planned setup

ItemDetails
BoardM5Stack CoreS3 Development Kit (K128)
Dev environmentWindows 11 + arduino-cli
LibrariesM5Unified 0.2.18 / M5GFX 0.2.25
ServerThe STT+LLM+TTS voice chat server (over Tailscale)
Filler audioWAVs pre-generated with Irodori-TTS
Lip sync imagesThree 128x128 PNGs (mouth closed / half open / open)

I checked the header of the WAVs the TTS server returns: 48kHz, mono, 16bit PCM.
The filler audio comes out in the same format, so playback format is the first thing to check.

Can it play filler WAVs from SD on the built-in speaker?

The M5Unified repo has a sample, examples/Advanced/Speaker_SD_wav_file, that implements exactly this: read a WAV from SD and play it on the built-in speaker.

The supported formats live in the validation code in Speaker_Class.cpp, so here is the excerpt.

// validation in Speaker_Class.cpp playWav() (excerpt)
if ( memcmp(wav->RIFF,    "RIFF",     4)
  || memcmp(wav->WAVEfmt, "WAVEfmt ", 8)
  || wav->audiofmt != 1          // linear PCM only
  || wav->bit_per_sample < 8
  || wav->bit_per_sample > 16    // 8-16bit
  || wav->channel == 0
  || wav->channel > 2 )          // mono / stereo
  { return false; }

The conditions are linear PCM, 8-16bit, up to 2 channels, and any sample rate.
The rate read from the header gets resampled in the background with linear interpolation to the output side (48kHz by default on the CoreS3), so no pre-conversion is needed.
The fillers are 48kHz mono 16bit WAV, so they will play.

Playback runs as a background FreeRTOS task, and playWav() returns immediately.
M5.Speaker.isPlaying() reports completion, and loop() stays free for screen drawing and HTTP while audio plays.
One caveat: do not free the buffer you passed until playback finishes. Free it early and, naturally, playback breaks.

Each filler is a few hundred KB, so instead of reading from SD every time, load them into PSRAM (8MB) at boot.
48kHz mono 16bit works out to about 94KB per second, so 8MB holds about 85 seconds.

struct WavClip { uint8_t* buf; size_t len; };
static WavClip filler[3];

bool loadWavToPsram(const char* path, WavClip& clip) {
  File f = SD.open(path);
  if (!f) return false;
  clip.len = f.size();
  clip.buf = (uint8_t*)heap_caps_malloc(clip.len, MALLOC_CAP_SPIRAM);
  size_t rd = f.read(clip.buf, clip.len);
  f.close();
  return clip.buf && rd == clip.len;
}

// playback is async; the call returns immediately
M5.Speaker.playWav(filler[0].buf, filler[0].len);

As covered in the earlier microSD troubleshooting, my card only accepts writes at 10MHz and will not remount after a soft reset until the power is cut.
For filler duty the card is only read at boot, so once everything finishes loading there, no further SD access is needed.

Can it record from the mic and send it to the server?

The CoreS3 has dual built-in mics (the ADC is an ES7210), recordable through M5Unified’s Mic class.
record() takes a buffer, an element count, and a sample rate; the default is 16kHz.
A background task fills the buffer, and isRecording() returns false once recording is done.

The mic and speaker share the same I2S bus and cannot run at the same time.
In M5Unified.cpp’s CoreS3 config, both are assigned to I2S_NUM_1 with BCK=GPIO34 and WS=GPIO33.
The official Microphone.ino sample does this switching explicitly.

while (M5.Mic.isRecording()) { M5.delay(1); }
M5.Mic.end();
M5.Speaker.begin();   // speaker usable from here

// switch back once playback ends
M5.Speaker.end();
M5.Mic.begin();

Recording, wrapping as WAV, and POSTing is an established pattern in the AI Stack-chan projects.
AI_StackChan2 allocates 3.75 seconds (about 120KB) of 16kHz 16bit mono in PSRAM, writes a 44-byte RIFF header directly at the head, records PCM behind it, and POSTs the whole thing as multipart/form-data to the Whisper API.
No filesystem is involved, so recording never needs to touch the SD.

// record 3.75s at 16kHz into PSRAM (AI_StackChan2's layout)
constexpr size_t record_size = 60000;   // 16000Hz x 3.75s
int16_t* buf = (int16_t*)heap_caps_malloc(record_size * 2, MALLOC_CAP_SPIRAM);
for (int i = 0; i < 400; i++) {
  M5.Mic.record(&buf[i * 150], 150, 16000);
}

My server also takes WAV as multipart, so pointing the same code at a different URL should send it as-is.
Qwen3-ASR on the STT side is fine with 16kHz WAV input.

As for the recording trigger: the CoreS3 has no physical buttons (even the ones below the screen are touch), so touch-to-start with a fixed recording length is what most implementations do.
Some examples end recording on silence detection, but fixed length is what I will implement first.

Another option would be keeping the mic always on and sending only after recognizing a wake word - here that would be calling out “Kana-chan” - but that is probably a stretch for the M5Stack’s horsepower.
Also on the audio side, the M5Stack ships with no physical volume keys or anything like that for playback. Getting blasted at full volume would be a problem, so I’ll need to figure out where volume adjustment goes too.

Can it lip-sync with 128x128 PNGs?

If the thing is going to talk, it should have a face, so I made pixel art of Kana-chan to put on screen.
The images are three 128x128 PNGs (mouth closed / half open / open), about 18KB each.

M5GFX has a built-in PNG decoder, and drawPngFile() can draw a PNG from SD directly.
But there are measured reports of ESP32-class PNG decoding taking a few hundred ms at 320x240, so decoding on every display may not keep up with the mouth switching rate (2-4 times per second).

Instead, decode once at boot into sprites (M5Canvas) and switch with pushSprite().
A 128x128 16bpp sprite is 32KB, or 96KB for all three.
That fits in internal RAM, never mind PSRAM, so DMA transfer is available too.
LovyanGFX’s rendering benchmarks put a full-screen 320x240 pushSprite at 30-50ms, so I expect the 128x128 switch to come in under that.

M5Canvas mouth[3] = { M5Canvas(&M5.Display), M5Canvas(&M5.Display), M5Canvas(&M5.Display) };
const char* pngs[3] = { "/close.png", "/half.png", "/open.png" };

void setup() {
  // after SD.begin()
  for (int i = 0; i < 3; i++) {
    mouth[i].setColorDepth(16);
    mouth[i].createSprite(128, 128);   // 32KB each
    mouth[i].drawPngFile(SD, pngs[i]); // decode only once at boot
  }
}

void loop() {
  M5.update();
  if (M5.Speaker.isPlaying()) {
    static bool open_mouth = false;
    mouth[open_mouth ? 2 : 0].pushSprite(96, 56);
    open_mouth = !open_mouth;
    M5.delay(150);
  } else {
    mouth[0].pushSprite(96, 56);
  }
}

The speaker is on I2S and the display on SPI - separate buses - so playback and drawing can run at the same time.

As a refinement, Stack-chan’s m5stack-avatar computes how open the mouth should be from the amplitude of the audio buffer being played and lip-syncs to that.
The three images would map onto that with two amplitude thresholds, so if fixed-interval switching looks off I will try changing to it, and if that still is not enough, add more in-between frames.

Bus layout and what to verify on hardware

Here is how the buses divide up when all three pieces run on one device.

BusConnectedConstraint
I2S (I2S_NUM_1)Speaker (AW88298) / Mic (ES7210)No simultaneous use. Switch with end() and begin()
SPILCD (ILI9342C) / microSDShared bus. No contention if SD loading finishes at boot
I2CMic/amp control / Power (AXP2101) / TouchManaged by M5Unified

Looking at the state transitions, the mic/speaker switch points are pinned to exactly two places.

flowchart TD
    A[Idle<br/>Mic on] -->|touch| B[Recording]
    B -->|fixed length ends<br/>Mic off / Speaker on| C[Send to server<br/>receive job ID]
    C --> D[Play filler<br/>lip sync]
    D -->|chunk arrives| E[Play reply<br/>lip sync]
    E -->|done<br/>Speaker off / Mic on| A

From this check, all three pieces hold up on spec.
What to verify on the hardware:

  • Whether my LAZOS 32GB card finishes loading all the filler WAVs and images at boot
  • How much the mic gain (magnification, default 16) and noise filter need adjusting
  • Noise or clipped audio around the Mic/Speaker switch (implementations insert a 200-500ms wait around it)
  • Whether fixed-interval lip sync looks natural
  • Where volume adjustment should live on the device
  • Whether to remove the CO2 sensor attached as a CO2 monitor, or keep it on and decide what it should do