M5Stack CoreS3 lip sync: 87ms sprite decode and why BtnA/B/C never fire

In the previous part the filler audio was playing from the SD cache with a volume UI. This time the device gets a face. The goal for this stage is Kana-chan’s pixel-art face lip-syncing while a filler plays; the actual talk-to-it part comes next.

The assets

I prepared three 128x128 PNGs: mouth closed (neutral), half open, and open — RGBA pixel art, about 18KB each.

I’m using the approach settled on paper during the spec check. PNG decoding on ESP32-class chips is slow (there are reports of a few hundred ms for a 320x240 image), so instead of decoding on every draw, decode once at boot into sprites (M5Canvas) and switch between them with pushSprite() afterwards.

A 128x128, 16bpp sprite is 32KB — 96KB for all three. That fits in internal RAM, never mind PSRAM, and DMA transfer works too. The speaker is on I2S and the display on SPI — separate buses — so playback and drawing can run at the same time.

PNGs go in flash, not on SD

The original plan had the PNGs on the SD card as well. That changed.

As the previous on-device test showed, the CoreS3’s LCD and SD share the same SPI bus through different drivers, and mishandling that causes mount failures and broken writes. For the fillers, the fix was to confine all SD work to a “no-drawing phase” right after boot.

The face PNGs would probably work if read in that same phase, but the face is fixed material — unlike the fillers, whose content comes from the server. At 56KB for all three, I turned the PNG binaries into C arrays and embedded them in the code.

# PNG → Cヘッダ化(xxd相当)
python -c "
data = open('kana_close.png','rb').read()
print('const unsigned char kana_close_png[] PROGMEM = {')
print(','.join(str(b) for b in data))
print('};')
" > kana_close.h

M5GFX (LovyanGFX) decodes a PNG straight from memory, so SD no longer needs to be involved.

mouth[i].setColorDepth(16);
mouth[i].createSprite(128, 128);
mouth[i].drawPng(kana_pngs[i], kana_png_lens[i], 0, 0);  // メモリから直接デコード

With this, SD’s only role is caching the filler audio, and the face shows up even with the card removed. No new SD reads got added.

Driving the lip sync

M5.Speaker.isPlaying() tells me whether audio is playing, so the first version uses fixed-interval switching.

// 再生中: 閉じ→半開き→開き→半開き→... を150ms間隔で回す
static const uint8_t seq[] = {0, 1, 2, 1};
if (M5.Speaker.isPlaying()) {
  mouth[seq[(millis() / 150) % 4]].pushSprite(FACE_X, FACE_Y);
} else {
  mouth[0].pushSprite(FACE_X, FACE_Y);  // 待機中は口閉じ
}

With three frames it’s not a binary closed↔open; the half-open mouth goes into the transitions for a 4-frame loop. If that looks off, the fallback is what Stack-chan’s m5stack-avatar does: derive the mouth opening from the amplitude of the audio buffer being played.

Separating face taps from UI controls

The default screen should be just the face. But if “tap to show the UI” exists, it collides with the end goal, “tap to talk.” If a touch flips the screen over to the UI, there’s no clear moment left to talk to the device.

The CoreS3’s only physical keys are the power button and reset, and the power button is already taken as power-off. So I decided to use the area below the display that isn’t used for drawing. The CoreS3’s touch panel responds below the LCD (240px), and in M5Unified that should be usable as the virtual buttons BtnA/B/C. It’s outside the LCD, so it can’t physically collide with face taps.

Here’s the assignment I had:

Input	Action
Face tap (on screen)	Talk (placeholder play/stop for now)
Below-screen BtnA (left)	Volume [-]; volume overlay shows for 1s then disappears
Below-screen BtnC (right)	Volume [+]; same
Below-screen BtnB (center)	Switch face screen ⇔ UI screen
Power button	Power off (as before)

The face screen puts the 128x128 face large in the center, with only the power status (battery/USB) shown small at the top. The volume buttons and status line from last time move to the UI screen, so normally only the face is visible. Volume can be adjusted from the below-screen touch without opening the UI screen, and the number overlays only at the moment it changes.

I also considered repurposing a short press of the power button for the UI switch, but didn’t want to erode the “press it and it turns off” reliability I had just fixed. …and with that designed, I went to the hardware.

On-device test

The area below the screen is outside the touch sensor

The first thing I checked after flashing was the below-screen touch — and it didn’t react anywhere. Printing raw touch coordinates over serial while touching that area made the cause plain.

touch press: x=297 y=239   ← y stops at 239 even when touching below the screen
touch press: x=319 y=239
touch press: x=181 y=238

Nothing past y=239 ever comes back. Touching below the LCD registers as “touched the very bottom of the screen,” and touch outside the LCD can’t be captured at all. I wondered if I had a defective unit, but it turned out to be how the CoreS3 is built. In the M5Unified source, the CoreS3 gets the same treatment as the Core2, where touches at y>=240 count as BtnA/B/C.

case board_t::board_M5StackCore2:
case board_t::board_M5StackCoreS3SE:
case board_t::board_M5StackCoreS3:
  tb_y = 240;   // y>=240のタッチをBtnA/B/C判定にする

On the Core2 this works because the touch film extends below the display and y=240–279 actually comes back. The CoreS3’s touch never reports past y=239; the strip holding the camera and microphones is outside the sensor, so a y>=240 touch can’t happen. The official docs describe the bottom mic/camera positions as the BtnA/B/C touch zones, but community testing reports “BtnA sometimes fires if you touch the bottom rim of the screen,” and that’s about it. I had assumed Core2-style buttons below the screen; on the CoreS3 that region never existed. Every tap aimed there was being taken by “on-screen tap = play/stop.”

M5Unified does have M5.setTouchButtonHeight(), which extends button detection up into the screen, so getting virtual buttons on a CoreS3 means giving up a strip at the bottom of the display. I tried making the bottom 24px a button bar (left = vol-, center = switch, right = vol+), and that failed too. 24px is about 2.4mm on the physical screen, and mis-taps piled up. I kept touching the face when I meant to press a button, and Kana-chan wouldn’t stop talking.

Split by gesture: tap, hold, flick

I stopped relying on button area and split the controls by gesture type.

Gesture	Action
Tap (anywhere)	Talk (pseudo-response for now = filler play/stop)
Hold (0.5s)	Switch face screen ⇔ UI screen
Vertical flick	Volume ± (overlay for 1s)

M5Unified’s touch already distinguishes clicks, holds and flicks, so it’s just a matter of branching on them.

auto t = M5.Touch.getDetail();
if (t.wasFlicked()) {
  if (abs(t.distanceY()) > abs(t.distanceX()) && abs(t.distanceY()) >= 30) {
    apply_volume(g_volLevel + (t.distanceY() < 0 ? 1 : -1), true);  // 上フリックで+
  }
} else if (t.wasHold()) {
  toggle_screen();
} else if (t.wasClicked()) {
  toggle_play();
}

Tap, hold and flick aren’t separated by location, so the bottom-bar style of mis-tap didn’t happen. “Touch the face and it talks” works as originally planned.

Lip sync in action

A tap plays a filler and the mouth moves only while it talks. Also switching to the UI screen with a hold and changing the volume with flicks.

Numbers, too:

• Decoding the 3 flash-embedded PNGs took 87ms (once, at boot)
• RGBA transparency baked onto the black background looks fine; no flicker when switching with pushSprite
• The 150ms, 4-frame loop (closed→half→open→half) looked right for the short filler interjections
• Filler playback (I2S) and lip-sync drawing (SPI) ran simultaneously without issues
• The SD-cache boot from last time still works (a few seconds to SDから読み込み ×3 → face shown, no WiFi)

Progress this session

• Sprites from the 3 flash-embedded PNGs (87ms decode)
• RGBA transparency rendering (fine baked onto black)
• pushSprite switching: flicker and speed
• Fixed-interval (150ms, 4-frame) lip sync appearance
• Filler playback and lip-sync drawing at the same time
• Control misfires → no below-screen touch, bottom-bar mis-taps, solved with gestures
• Volume overlay coexisting with the lip sync
• SD-cache boot still working

Planned for next session

From here there are three hard parts — recording, networking, and the async assembly — so it’s split into one hard part per stage. Next is recording on its own.

• A voice recorder that records a fixed number of seconds on touch, then plays it back (no server involved)
• Switching the Mic/Speaker I2S exclusivity (end()/begin(), noise and cut-offs around the switch)
• Tuning the mic gain (magnification)
• Recording 48kHz/16bit/mono into PSRAM