StackChan CoreS3 Voice Chat: ESP-IDF Build & Tailscale Direct UDP
Contents
In my previous test, running Tailscale direct UDP on a standalone CoreS3 test program dropped the average WAV download time from the voice server to about 2.4 seconds.
That was a significant improvement from the roughly 9 seconds over Tokyo DERP.
However, the main voice chat running on the StackChan Body was still routing through a PHP relay on a VPS.
The test program was built with ESP-IDF, while the voice chat was written in Arduino style. Because the build systems were completely different, I decided to port the voice chat to ESP-IDF and integrate Tailscale directly to talk without the VPS relay.
Test Environment
| Item | Environment |
|---|---|
| Device | M5Stack CoreS3 (ESP32-S3, 16MiB Flash, 8MiB PSRAM), StackChan Body |
| Host PC | Windows, Tailscale 1.102.4 |
| Build | ESP-IDF 5.5.4, Docker image espressif/idf:v5.5.4 |
| Arduino Core | arduino-esp32 3.3.10 (as an ESP-IDF component) |
| Libraries | M5Unified 0.2.17, M5GFX 0.2.24, StackChan-BSP 1.1.0 |
| Server | Local voice server at home (same as before) |
Why Tailscale Is Hard to Add to Pure Arduino Builds
The voice chat sketch was originally compiled using arduino-cli with esp32 core 3.3.10.
An Arduino build runs on top of a pre-compiled ESP-IDF static configuration (sdkconfig).
The Tailscale components I patched previously depend on low-level lwIP and TLS configurations.
Specifically, lwIP core lock (mutual exclusion) and TLS buffer allocations cannot be altered in a pre-compiled Arduino sdkconfig.
I reversed the architecture: I built the entire project under ESP-IDF and embedded Arduino as an ESP-IDF component (arduino-esp32).
This is an officially supported Espressif approach, and the existing voice chat code ran virtually unchanged.
Converting the Voice Chat into an ESP-IDF Project
I renamed .ino to main.cpp and added #include <Arduino.h> at the top.
The headers containing embedded facial expressions and standby audio clips, as well as the 6MB app partition table, were reused as-is.
The libraries were placed inside the project’s components/ directory:
| Library | Integration Method |
|---|---|
| M5Unified 0.2.17, M5GFX 0.2.24 | Copied from local Arduino libraries. Enabled the conditional build lines in CMakeLists.txt for arduino-esp32. |
| StackChan-BSP 1.1.0 | Wrote a minimal CMakeLists.txt since it had no native ESP-IDF support. IR and NFC dependencies were ignored. |
| Tailscale, WireGuard | Copied from the previous test project. |
The board configuration matched the compiler flags used by arduino-cli for CoreS3:
USB serial assignments (ARDUINO_USB_MODE=1, ARDUINO_USB_CDC_ON_BOOT=1) and PSRAM were enabled with QSPI mode.
Build Errors Under ESP-IDF 6.0
Since the previous standalone test project used ESP-IDF 6.0, I initially tried building under 6.0.
On paper, arduino-esp32 3.3.10 supports ESP-IDF versions from 5.3 up to 6.1.
| Failure Point | Issue | Workaround |
|---|---|---|
| CMake | The component version of arduino-esp32 lacked CoreS3 pin definitions | Used the generic ESP32-S3 pin definition since neither my code nor libraries used CoreS3 pin definitions |
| Compilation | 6.0 headers threw C++ warnings treated as errors by default | Disabled warning-as-error in sdkconfig |
| arduino-esp32 SD library | Incompatible with updated FAT APIs and types in 6.0 | Disabled unused SD libraries in component configuration |
| M5Unified | 6.0 changed I2S port definitions, breaking compilation | Fixing one led to cascading errors elsewhere |
By default, the component configuration enables all bundled Arduino libraries. I had to explicitly disable 26 unused libraries (such as SD) to get arduino-esp32 itself to build.
Upgrading M5Unified to 0.2.23 resolved the I2S syntax, but 0.2.23 also changed microphone and speaker handling, including recording adjustments that could disrupt my tuned voice capture settings.
Looking at the pre-compiled esp32 core 3.3.10 libraries in my local arduino-cli, they were actually built against ESP-IDF 5.5.4.
Since Arduino and M5Unified were already proven to work reliably under 5.5.4, I decided to downgrade the entire project to ESP-IDF 5.5.4 and adjust the Tailscale components instead.
Getting It to Build Under ESP-IDF 5.5.4
The Tailscale components only rely on ESP-IDF TLS, certificate bundles, lwIP, and the HTTP client, without calling low-level crypto libraries directly.
Dropping a single 6.0-specific component requirement allowed Arduino, M5Unified, and StackChan-BSP to build cleanly.
| Failure Point | Cause | Workaround |
|---|---|---|
| Tailscale | 6.0 builds C code as C23 (where bool is built-in), whereas 5.5 defaults to C17 | Added <stdbool.h> to three headers using bool |
| Voice chat main | WiFi.h from arduino-esp32 could not be found | Added Wi-Fi and network component dependencies to the main component |
| Voice chat main | WiFi.h tried to include disabled Wi-Fi provisioning headers | Re-enabled that specific Arduino library component |
| Linker | Missing HTTPS client symbols | Enabled PSK (Pre-Shared Key) settings in mbedTLS configuration to match Arduino |
Checking Arduino’s default esp32s3 configuration, lwIP core lock was also enabled.
As discovered in the previous test, receiving direct UDP packets locked up lwIP unless the double-lock bug was patched, so that patch remained essential here.
The final binary came out to 4,113,680 bytes, very close to the arduino-cli build size (4,183,715 bytes).
Verifying Behavior Against the Original Build
First, I flashed the firmware without starting Tailscale.
All 5 facial expressions, saved volume settings, Wi-Fi connectivity, and baseline touch calibration initialized exactly like the original Arduino version.
I verified conversation handling over serial.
recnow 3 is a test command that records for 3 seconds and runs one conversation turn. With nobody speaking, it is expected to return no_speech.
The CoreS3 uploaded 96,187 bytes to the VPS relay, received HTTP 200 in 782 ms, and polled back the expected no_speech response.
Switching the underlying build system did not break conversation over the relay.
Adding the Boot Sequence for Direct UDP
Immediately after boot, Wi-Fi may disconnect and reconnect, and the initial handshake with the voice server takes several seconds.
Speaking to the robot during that window causes timeouts or errors, so I suppressed the facial display and blocked user input until everything is ready.
| Stage | Condition | Timeout |
|---|---|---|
| Wi-Fi | Connected continuously for 3 seconds | 30s |
| Time Sync | Valid NTP timestamp acquired | 30s |
| Tailscale | Node authorized and assigned a Tailscale IP | 120s |
| Voice Server | Health check URL returns 200 (also completes WireGuard handshake) | 60s |
The display shows “Booting…” and the current stage.
If any stage fails, the device falls back to VPS relay mode.
Because I reused the same Tailscale node name, it rejoined the tailnet using credentials stored in NVS flash without requiring browser re-authorization.
On the very first boot, serial output froze during the voice server check.
Pinging the CoreS3 from my PC via tailscale ping confirmed that direct UDP packets were answering normally—the system itself was still running.
The freeze happened because ESP-IDF log output was directed to the same USB serial CDC port as Arduino’s Serial.
Following the Arduino CoreS3 preset, I routed ESP-IDF logs to UART0 and kept USB CDC as a secondary sink, which eliminated the contention.
Subsequent boots completed all 4 stages in 26.8 seconds into direct connection mode.
Running recnow 3 over direct UDP returned HTTP 200 in 1,580 ms on the first run and 474 ms on the next. Since both were single measurements, variance cannot be distinguished from relay performance yet.
Wi-Fi Timeout Handling
On one boot attempt, Wi-Fi failed to connect within 30 seconds, triggered fallback to VPS relay, and then finally reconnected 30 seconds later.
A similar 30-second Wi-Fi timeout had occurred once back when developing the Arduino version.
Since the relay also requires Wi-Fi, dropping to relay on Wi-Fi failure did not help.
I updated the boot sequence to retry Wi-Fi reconnection up to 3 times before giving up.
Touch Sensor False Triggers
While running in direct UDP mode, the capacitive touch sensor on the head repeatedly misdetected taps.
This repeatedly triggered recordings that ended in “Could not hear you.”
During testing, I had not touched the robot at all.
Every detected tap in the log was a false trigger.
| Boot Attempt | Connection | Log Duration | Tap Triggers |
|---|---|---|---|
| Build verification | VPS Relay | ~45s | 0 |
| Direct UDP #1 | Direct | ~2.5 min | 1 |
| Wi-Fi fallback | VPS Relay | ~2.5 min | 0 |
| Direct UDP #2 | Direct | ~4 min | 18 |
| Reverted to Arduino | VPS Relay | ~35s | 2 |
Initially it seemed like a Tailscale-specific issue, but reverting to the original Arduino build also yielded 2 false taps within 35 seconds.
The relay logs had simply been too short to catch it; the issue was not caused by ESP-IDF or Tailscale.
The robot sat in open air with nothing nearby.
Even so, raw touch readings hovered between 1 and 2, occasionally spiking to 3.
Since a tap is registered whenever intensity reaches 3 or higher, those spikes triggered recording.
Across all 19 occurrences during the two direct boots, the robot waited 5 seconds without voice input and discarded the buffer without uploading to the server.
The screen spent 6 seconds displaying “Recording…” followed by “Could not hear you.”
Looking at the 10 readings immediately before each trigger, 6 to 8 showed touch presence.
Past logs of intentional taps showed a similar 7 to 8 readings. They were indistinguishable from real taps based solely on pre-trigger values, and raising the threshold would make genuine taps unresponsive.
Instead, I shortened the silence timeout from 5 seconds to 2.5 seconds, hid the UI on silence discards, and added a boot check that waits until touch readings stay at 0 for 3 consecutive seconds.
Even with that, 26 false triggers fired over 3 minutes starting 6 seconds after face display.
However, after the last trigger, not a single false tap occurred for over 15 minutes.
The previous boot had shown a similar pattern of false triggers stopping after about 3.5 minutes.
Because I was rebooting after every flash, I had only been observing the first few minutes of operation.
The noise appears concentrated immediately after boot and subsides over time, though the physical reason remains unconfirmed.
Suppressing Touch Triggers During Warm-Up
After showing the face, I added a rule that ignores all taps until touch readings remain 0 for 30 consecutive seconds.
If any touch is registered, the 30-second counter resets. If it does not settle after 5 minutes, input is accepted anyway.
While waiting, “Warming up” is shown at the bottom of the display.
In my initial implementation, the sensor failed to respond to real taps even after “Warming up” disappeared.
Logs showed readings maxing out at intensity 2.
This happened because the sensor was recalibrating its baseline every 15 seconds even while false readings were active.
Calibrating during active noise shifts the baseline upward—as a result, real taps registered too weakly. This was a known quirk from the Arduino build, so I restricted recalibration to periods when no touch is registered.
| Stage | Duration |
|---|---|
| Boot to voice server direct connection | 28.5s |
| Waiting for touch readings to clear | 23.4s |
| Warm-up suppression (ignored 48 false triggers) | 91.3s |
From power-on to accepting user taps took about 2.5 minutes.
A Complete Conversation Turn Over Direct UDP
Once the warm-up indicator cleared, tapping the head registered at intensity 3 and started recording.
Voice audio was sent directly over Tailscale UDP, and the response was played back.
| Segment | Measurement |
|---|---|
| Recording | 3,803 ms (1,100 ms until speech began) |
| Audio upload (57,787 bytes) | 331 ms |
| Time to emotion response | 10,088 ms |
| Time to sentence 1 audio | 14,068 ms |
| Sentence 1 audio download (170,284 bytes) | 771 ms |
| Sentence 1 playback start | 14,897 ms |
| Sentence 2 audio download (101,164 bytes) | 587 ms |
| Total turnaround time | 23,782 ms |
Downloading approximately 170KB (170,284 bytes) took 771 ms (about 216 KiB/s), matching the standalone test program.
Most of the ~15-second delay before sentence 1 playback was server-side LLM and TTS generation time.
During this test turn, audio did not play aloud because a volume setting of 0—sent earlier to silence false triggers—remained saved in flash.