M5Stack CoreS3 Tailscale: Direct UDP Drops WAV Download from 9s to 2.4s
Contents
Update (2026-09-25): Ported the StackChan voice chat sketch to ESP-IDF and integrated this Tailscale direct UDP setup to connect directly to the home voice server → StackChan CoreS3 Voice Chat: ESP-IDF Build & Tailscale Direct UDP
Last time, I connected an M5Stack CoreS3 to a Tailscale network to fetch WAV audio directly from a self-hosted voice server without going through a VPS proxy.
However, the traffic was pinned to a Tokyo DERP relay server and took roughly 9 seconds to download a 433,964-byte WAV file at about 47 KiB/s.
Fetching the exact same file from a PC on the same tailnet completed in just 0.154 seconds.
Tailscale typically discovers peers via DERP and then upgrades the connection to direct UDP.
During the previous test, this direct path negotiation failed, and I had to fall back to DERP-only mode.
I dug into the implementation to see if the CoreS3 could establish a direct, relay-free UDP connection.
Test Environment
| Item | Details |
|---|---|
| Device | M5Stack CoreS3, ESP32-S3, 16MiB Flash, 8MiB PSRAM |
| Workstation | Windows, Tailscale 1.102.4 |
| Build Toolchain | ESP-IDF 6.0, Docker image espressif/idf:v6.0 |
| Flashing Tool | esptool 5.3.0 |
| Destination | Self-hosted voice server on a separate internet connection |
| Test Payload | Same standby WAV as previous test, 433,964 bytes |
Checking Direct Connectivity Between PCs
First, I verified the network path between my workstation and the voice server.
tailscale ping -c 5 laptop-0e6caiut
pong from laptop-0e6caiut (...) via <server-public-ip>:<port> in 56msThe via field returned the remote peer’s public IP rather than a DERP relay. This confirmed direct UDP communication right from the start.
The PC and the voice server sit behind entirely distinct external IP addresses on separate lines.
Running tailscale netcheck reported that UDP was functioning properly and NAT port mappings were endpoint-independent (MappingVariesByDestIP: false).
The nearest DERP relay was Tokyo at 10.1 ms latency.
Between standard Tailscale clients on my home network, direct connections form without issues.
If only the CoreS3 falls back to relaying, the problem is on the ESP32 side.
Discrepancies Between Borrowed DISCO and Official Tailscale
Tailscale uses a mechanism called DISCO (Discovery) to establish direct paths. It sends Ping probes to candidate addresses and switches to direct routing once a Pong is received.
The serial_wifi_logger commit c537f4b used in my previous setup already included an initial DISCO implementation.
The same author’s portable_terminal repository contains an expanded version along with porting notes.
Comparing the code against these notes revealed that the DISCO implementation in c537f4b did not conform to official Tailscale packet formats.
| Attribute | c537f4b | Official Tailscale (from porting notes) |
|---|---|---|
| Encryption | XChaCha20-Poly1305 | NaCl crypto_box (XSalsa20-Poly1305) |
| Packet Header | 6-byte magic, 24-byte nonce | 6-byte magic, 32-byte sender DISCO pubkey, 24-byte nonce |
| Ping Payload | Type, 8-byte ID | Type, version, 12-byte ID, sender node key |
| UDP Socket | Dedicated DISCO socket | Shared with WireGuard socket |
| Incoming Pings | Ignored | Responds with Pong |
| Endpoint Advertising | Sends empty list | Advertises public IP and port |
Under this mismatch, remote tailscaled instances cannot parse incoming Pings.
The timeout during my earlier direct connection attempt likely happened because the code switched WireGuard’s endpoint to the candidate address prematurely without waiting for a Pong.
Replacing with portable_terminal’s Implementation
I pulled components/tailscale and components/wireguard from portable_terminal commit 44a616e.
According to its porting notes, this version rewrites DISCO, implements a graceful transition from DERP to direct paths, probes all candidate endpoints, fetches and advertises external endpoints via HTTPS, and handles CallMeMaybe messages (requests asking a peer to send a probe).
I carried over two bug fixes from my previous test:
| Fix | Description |
|---|---|
| Interactive Auth Re-registration | Sends previous URL via Followup during browser login, omitting netmap requests until authenticated |
| DERP TLS Synchronization | Protects each TLS read/write pair with a shared mutex; maintains non-blocking I/O post-connection |
portable_terminal’s DERP handler previously locked transmissions while reading without synchronization.
Because this concurrency clash crashed mbedTLS in my earlier tests, I wrapped both operations in the mutex.
Since portable_terminal allocates buffers such as the netmap in external PSRAM, I re-enabled PSRAM, which had been disabled in the earlier probe.
I also removed configurations that forced DERP-only mode and fixed relay routing to Tokyo.
The test app was set up to continuously fetch the same WAV file every 15 seconds after booting, and recorded download duration on each attempt to observe the transition from DERP to direct UDP.
First Boot: WAV Download Hangs
Reusing saved node keys allowed the CoreS3 to rejoin the tailnet without browser re-authentication.
Around 32 to 35 seconds after startup, the CoreS3 began receiving Pong responses to its Ping probes:
I (34667) ts_disco: Pong recv from <server-public-ip>:<port> → peer 12 direct
I (44857) wireguard: [WireGuard] HANDSHAKE_RESPONSE: e8c68499:<port>Peer 12 was the voice server, matching the exact public IP and port observed via workstation tailscale ping.
The WireGuard handshake response arrived from that same public IP.
If routed via DERP, this would have appeared as the pseudo-address 127.3.3.40 (logged as 2803037f).
However, the WAV download never completed.
Despite a 15-second HTTP timeout setting, execution hung indefinitely for over 5 minutes without throwing errors.
Both DERP transmission tasks froze, and send queues began overflowing around the 310-second mark.
Given a 16-slot queue and 15-second keepalive intervals, the stall must have occurred around second 46.
Meanwhile, the CoreS3 continued responding to incoming UDP Pings from the server; the device had not completely locked up.
Finding the lwIP Double Locking Bug via Backtrace
I added a watchdog that dumps backtraces across all tasks whenever a single fetch exceeds 45 seconds, then rebooted the device.
Once the hang recurred, I resolved the instruction pointers to function names using addr2line.
| Task | Blocked Location |
|---|---|
| tcpip (lwIP processing thread) | LOCK_TCPIP_CORE() inside wireguardif_network_rx called from udp_input |
| main (WAV fetch) | Inside lwip_select during HTTP connection |
| DERP TX (x2) | Waiting on lwIP lock during TLS write |
| DERP RX (x2) | Waiting on TLS mutex held by TX task |
lwIP is the network stack; packets arriving from Wi-Fi are processed on its dedicated thread.
This thread holds the global lwIP core lock throughout packet processing.
When WireGuard packets arrived over direct UDP, wireguardif_network_rx attempted to acquire LOCK_TCPIP_CORE() a second time.
Because this lock is non-recursive, the thread deadlocked against itself.
Every other task requiring the network stack subsequently stalled behind it.
Incoming DISCO Ping/Pong packets branched off before reaching this code path, which explained why discovery replies still functioned.
The lockup coincided precisely with the arrival of the very first direct WireGuard data packet.
Why this never triggered in portable_terminal remains unclear.
Its configuration lacked lwIP custom options; if core locking defaulted to disabled, that lock macro would have compiled to a no-op.
The CoreS3 project, however, explicitly enabled CONFIG_LWIP_TCPIP_CORE_LOCKING.
Packets received via DERP are also handed off to the processing thread before invoking the same callback, so this inner lock was redundant under both paths.
I eliminated the lock call in wireguardif.c.
Speed Improved, but Throughput Drops to 8 KiB/s
Removing the lock brought success on the third boot:
| Run | Time | Throughput |
|---|---|---|
| 1 | 10.017 s | 42.3 KiB/s |
| 2 | 3.215 s | 131.8 KiB/s |
| 3 | 2.187 s | 193.8 KiB/s |
| 4 | 48.346 s | 8.8 KiB/s |
| 5 | 53.823 s | 7.9 KiB/s |
| 6 | 48.801 s | 8.7 KiB/s |
On run 1, receiving HTTP response headers took about 8 seconds due to initial WireGuard handshake negotiation, while payload transfer took roughly 2 seconds.
By run 3, throughput had quadrupled compared to the previous DERP baseline.
Starting from run 4, headers still arrived within 1 second, but payload throughput plunged to around 8 KiB/s without any logged route changes.
Examining the logs revealed that incoming Pings from other tailnet nodes abruptly ceased around 90 seconds after boot (counts per 30-second window dropped: 49, 54, 4, 0).
Furthermore, public endpoint announcements were entirely absent.
While the CoreS3 appeared as a direct source to peers it explicitly probed, its advertised endpoint in the netmap remained empty.
Remote tailscaled daemons invalidated the direct route to the CoreS3, and return traffic from the voice server reverted to DERP.
Endpoint discovery begins after the second netmap streaming request to the coordination server.
Reviewing the earlier backtrace showed that this second request stalled waiting for data frames after receiving :status 200.
Tailscale communicates with coordination servers over HTTP/2, where receivers send WINDOW_UPDATE frames to grant flow control credit.
Under RFC 9113 section 6.9.2, the default initial window size is 65,535 bytes.
The CoreS3 client never sent flow control credit updates.
My tailnet contains 16 active nodes; once the initial netmap payload consumed the 64 KB window, the server could not deliver subsequent updates.
I patched the client to issue WINDOW_UPDATE frames matching the length of received DATA frames.
Throughput Collapses Again Around 200 Seconds
On the fourth boot, the second netmap response completed and public endpoint discovery began.
The first STUN server failed, but the fallback (checkip.amazonaws.com) resolved the public IP and advertised it successfully.
Around 86 seconds in, the device sent CallMeMaybe messages to 10 peers lacking direct paths.
Running tailscale ping from my workstation connected directly to the CoreS3 via its local LAN IP:
| Run | Time | Throughput |
|---|---|---|
| 1 | 16.473 s | 25.7 KiB/s |
| 2 | 2.399 s | 176.6 KiB/s |
| 3 | 1.695 s | 250.0 KiB/s |
| 4 | 2.752 s | 153.9 KiB/s |
| 5 | 1.948 s | 217.5 KiB/s |
| 6 | 1.754 s | 241.6 KiB/s |
| 7 | 2.108 s | 201.0 KiB/s |
| 8 | 3.609 s | 117.4 KiB/s |
| 9 | 3.933 s | 107.7 KiB/s |
| 10 | 2.561 s | 165.5 KiB/s |
| 11 | 48.978 s | 8.7 KiB/s |
| 12 | 50.800 s | 8.3 KiB/s |
The 90-second stall was resolved, but transfer rates dropped back to ~8 KiB/s starting on run 11 (212 seconds after boot).
Around second 205, direct Pings from the voice server stopped.
From second 212, the server began issuing CallMeMaybe frames via DERP every 5 seconds to ask the CoreS3 to send UDP probes and re-establish the direct link.
The CoreS3 failed to send a single Ping in response.
DISCO maintains a 64-entry table tracking pending Ping probes awaiting Pongs.
Entries in this table only clear when a matching Pong arrives.
Nodes on my tailnet include unreachable internal Docker bridge IPs and offline devices.
Unanswered probes accumulated until all 64 slots filled up after 83 total transmissions.
Subsequent probe attempts were discarded silently after emitting debug logs.
I modified the tracker to reap entries older than 5 seconds and emit warnings when the table fills up.
Additionally, the CoreS3 had selected Region 1 (New York) as its home DERP relay.
It picked the lowest region ID in the list—the same issue encountered in the previous test.
When direct UDP connectivity dropped, return traffic from the voice server routed through New York. This explained the sluggish ~8 KiB/s throughput.
While keeping direct UDP enabled, I pinned the preferred home DERP to Tokyo (Region 7).
Internal heap headroom shrank from ~220 KB at boot to ~128 KB after the first transfer, and stayed stable thereafter without leaking.
30 Consecutive Successful Direct UDP Transfers
The fifth boot ran 30 consecutive download iterations.
The home relay was properly set to Tokyo, and public IP advertisement succeeded.
| Run | Time | Throughput |
|---|---|---|
| 1 | 11.548 s | 36.7 KiB/s |
| 2 | 2.747 s | 154.3 KiB/s |
| 3 | 2.414 s | 175.5 KiB/s |
| 4 | 2.741 s | 154.6 KiB/s |
| 5 | 1.852 s | 228.7 KiB/s |
| 6 | 1.924 s | 220.2 KiB/s |
| 7 | 1.967 s | 215.4 KiB/s |
| 8 | 3.190 s | 132.8 KiB/s |
| 9 | 3.565 s | 118.9 KiB/s |
| 10 | 2.849 s | 148.7 KiB/s |
| 11 | 1.958 s | 216.4 KiB/s |
| 12 | 1.712 s | 247.5 KiB/s |
| 13 | 2.173 s | 195.0 KiB/s |
| 14 | 2.042 s | 207.5 KiB/s |
| 15 | 3.164 s | 133.9 KiB/s |
| 16 | 2.632 s | 161.0 KiB/s |
| 17 | 3.315 s | 127.8 KiB/s |
| 18 | 2.047 s | 207.0 KiB/s |
| 19 | 1.942 s | 218.1 KiB/s |
| 20 | 1.946 s | 217.7 KiB/s |
| 21 | 1.854 s | 228.5 KiB/s |
| 22 | 2.138 s | 198.2 KiB/s |
| 23 | 2.444 s | 173.4 KiB/s |
| 24 | 2.762 s | 153.4 KiB/s |
| 25 | 3.246 s | 130.5 KiB/s |
| 26 | 1.699 s | 249.3 KiB/s |
| 27 | 2.034 s | 208.3 KiB/s |
| 28 | 2.038 s | 207.9 KiB/s |
| 29 | 1.882 s | 225.2 KiB/s |
| 30 | 2.551 s | 166.1 KiB/s |
All 30 downloads succeeded without falling back to 8 KiB/s.
The DISCO probe table never overflowed.
Direct UDP Pings from the voice server arrived reliably at approximately 20 packets per minute over 9 minutes.
All 8 WireGuard handshakes with the voice server exchanged directly via its public endpoint.
An additional 40 handshakes traversed DERP for communications with other background peers.
| Path | Transfer Duration | Throughput |
|---|---|---|
| Previous Baseline (CoreS3 via Tokyo DERP) | 8.816 – 9.025 s | ~47 KiB/s |
| Current Setup (CoreS3 via Direct UDP, runs 2–30) | Avg 2.373 s, Median 2.138 s (1.699 – 3.565 s) | Avg 187.0 KiB/s |
| Workstation Benchmark | ~0.154 s | — |
Direct UDP quadrupled download speeds compared to DERP, though it still trails desktop performance.
Run 1 required 11.548 seconds due to initial WireGuard handshake negotiation right after boot.
During runs 1 to 3, Wi-Fi disconnected and re-associated twice between seconds 14 and 17, an anomaly that did not recur during runs 4 and 5.