Long-term memory for StackChan: 0.45 threshold drop on weekend plans
Contents
Following our previous test, we established the spec for adding recency scoring to retrieved memories and passing them to the prompt.
However, that benchmark only evaluated 16 handwritten facts and 10 questions on a desktop script, without generating memories dynamically from live conversations.
This time, I integrated long-term memory (vector retrieval of past conversation facts) directly into the StackChan voice chat server to test whether spoken details could be recalled through voice.
Test Environment
| Item | Details |
|---|---|
| Server PC | Production machine (Windows 11 Home, AMD Ryzen 7 5800HS, RTX 3050 Ti Laptop 4GB, 16GB RAM) |
| Speech Recognition (STT) | Qwen3-ASR-0.6B, CPU |
| Chat LLM | ModelScope API-Inference, Qwen-Ambassador/Qwen3.8-Max, enable_thinking: false, web search enabled |
| Embedding | Qwen3-Embedding-0.6B, 256 dimensions, local CPU, bf16 (16-bit floating point) |
| Vector Search | Qdrant local mode |
| Speech Synthesis (TTS) | Irodori-TTS, GPU |
Following our Omni-Flash test, the server pipeline was reverted to running transcription first.
If audio is fed directly into Omni-Flash, transcription text is not available before the reply begins, making it impossible to construct queries for memory retrieval. Long-term memory is therefore only active under this transcribe-first architecture.
Switching the Chat Model to Qwen3.8-Max
Before deploying the pipeline, I tested 6 questions once each with 3.7-Plus and 3.8-Max using the system prompt for Kana (the conversational persona for StackChan) with web search enabled.
| Question | 3.7-Plus First Token / Total | 3.8-Max First Token / Total |
|---|---|---|
| I’m home! Super tired today. | 2.3s / 2.6s | 1.5s / 1.9s |
| I lost my wallet… | 1.5s / 2.0s | 1.7s / 2.3s |
| What’s the weather in Tokyo tomorrow? (Web search on) | 2.9s / 3.5s | 1.8s / 2.3s |
| Want to play video games this weekend? | 1.7s / 2.2s | 1.9s / 2.5s |
| Stayed up late again last night… | 1.9s / 2.2s | 1.8s / 2.5s |
| What is the tallest mountain in Japan? | 3.7s / 4.1s | 2.1s / 2.8s |
Time to first token was faster with 3.8-Max on 4 out of 6 questions, shaving 1.1 seconds off the weather query and 1.6 seconds off Mt. Fuji. Web search only triggered for the weather query, increasing prompt size to 3,327 tokens. Since this was a single sample per question, latency variations should be taken as reference values.
To examine reply length and tone, three representative responses are shown below.
| Question | 3.7-Plus | 3.8-Max |
|---|---|---|
| I lost my wallet… | Oh no, that’s terrible! First, file a report with the police and immediately suspend your cards. | Oh, that must be stressful, are you okay? First, call the places you visited today and ask. |
| Want to play video games this weekend? | Sounds great, what should we play? I’m already excited! | Sounds fun! Today is Sunday, so we could even play right now. What game were you thinking of? |
| Stayed up late again last night… | Jeez, again? Make sure to go to sleep early tonight. | You stayed up late again? That’s bad for your health, so get to bed early! Just because tomorrow is Sunday doesn’t mean you can slack off. |
Responses from 3.8-Max were slightly longer.
In the late-night reply, 3.8-Max said “tomorrow is Sunday” even though the test day was Sunday. In the gaming query, it correctly identified Sunday; the model appears to have miscalculated tomorrow’s weekday on its own.
Since the system prompt only contained the current date and time, I updated it to supply tomorrow’s date and day of the week as well.
Integration Architecture
In a single turn, the only addition is inserting a vector memory retrieval immediately after speech transcription.
Memory extraction runs asynchronously on a separate thread. When conversation pauses for 5 minutes or reaches 8 turns, prior dialogue logs are sent in batch to Qwen to extract fact sentences.
flowchart TD
A[Receive Audio] --> B[Silence Detection]
B --> C[Transcribe with Qwen3-ASR]
C --> D[Retrieve Memory from Qdrant]
D --> E[Send Memory + Last 3 Turns to Qwen3.8-Max]
E --> F[Synthesize TTS Sentence by Sentence]
E -.Accumulate Logs.-> G[Extract Facts after 5m Silence or 8 Turns]
G --> H[Deduplicate and Save to Qdrant]| Parameter | Value |
|---|---|
| Short-term Context | Last 3 turns. Included in prompt history if last utterance occurred within 10 minutes |
| Extraction Trigger | After 5 minutes of silence or when 8 turns accumulate |
| Deduplication | Skipped if similarity with existing stored memory is 0.92 or higher |
| Retrieval | As established previously: top 8 candidates by raw cosine similarity, add 0.05 × exp(−elapsed_days / 14), and select up to top 3 above 0.45 threshold |
| Prompt Format | Instruction-guided Format B: wrapped in <retrieved_memory> with rules against unprompted disclosure, favoring recent dates, and avoiding false certainty |
Short-term 3-turn context and long-term memory serve distinct roles. Immediate continuity within 10 minutes is handled by chat history.
Long-term memory is used to retrieve facts pushed out of the 3-turn window or from previous days.
Fact Extraction
The extraction prompt targets user preferences, habits, plans, and events, while explicitly excluding Kana’s own statements, general knowledge, weather, news, sensor values, and casual greetings.
Output is structured JSON containing 1 sentence per fact, roughly 30 to 60 Japanese characters.
The subject is standardized to “you” (in Japanese, anata). In our previous 10-question test, when the memory subject was “user”, Kana began addressing the user as “User-kun”. Standardizing to “you” aligns with Kana’s system prompt, which instructs the persona to address the speaker as “you” rather than using placeholders.
Changing the subject also influences embedding similarity scores. I compared “user” vs. “you” across the 16 facts and 10 queries from previous tests, using September 10 as the baseline with recency bonus applied.
| Metric | ”user" | "you” |
|---|---|---|
| 10-query pass/fail | Passed 6 target/contradiction queries, filtered all 4 negative queries | Identical |
| Top-1 score on targets/contradictions | 0.475 - 0.648 | 0.524 - 0.645 |
| Max score on 4 negative queries | 0.434 (N1: Today’s plans) | 0.446 (N4: Lottery) |
| Max score on Lottery query (N4) | 0.389 | 0.446 |
Overall classification results remained identical. However, while target scores increased by ~0.03 (except for H4’s Python question), the negative lottery query (N4) also rose from 0.389 to 0.446, leaving only 0.004 of headroom below the 0.45 threshold.
For relative temporal expressions like “tomorrow” or “next Saturday”, the extractor was instructed to preserve the original phrase while appending the concrete calendar date in parentheses, such as “next Saturday (October 3, 2026)”.
Switching Embeddings to bf16
Because the embedding model remains resident in host CPU RAM, I converted model loading from fp32 to bf16.
I evaluated fp32 vs. bf16 across the 16 facts and 10 questions using the “you” subject.
| Metric | fp32 | bf16 |
|---|---|---|
| RAM increase after model load | 2381 MB | 1212 MB |
| Batch embed time (16 facts + 10 questions) | 2.9s | 15.6s |
| Single query embed time | 0.18s | 0.25s |
| 10-query pass/fail | Identical | Identical |
Cosine similarity between fp32 and bf16 vectors was at least 0.9970 across all items, with a maximum score shift of 0.005.
Although batch embedding was 5x slower, live runtime only embeds a single question query per turn, incurring an overhead of only 0.07 seconds. I chose bf16 to cut RAM footprint by half.
I also added an automatic fallback to ModelScope’s embedding API for when it recovers from earlier TTS resource conflicts. To avoid corrupting vector consistency with existing vectors, switching to the API only occurs if the API responds successfully and vectors produced for the same test string achieve a similarity of 0.99 or higher with local embeddings. During this test run, it remained on local execution.
Voice Query and Recall Test
For conversational input, I synthesized audio clips with TTS and streamed them to the server in place of physical StackChan microphone input.
In the first phase, I spoke 5 conversational utterances, triggered immediate memory extraction via a debug endpoint without waiting for the timeout, and then asked 5 recall questions in the second phase.
Because the two phases occurred within 10 minutes, the 3-turn short-term history remained active during testing.
The pipeline was run multiple times while refining extraction prompts. The table below shows the results from the improved prompt run.
First Phase: 5 Initial Utterances
Kana’s facial expressions reflect 5 emotion tags (joy, fun, smile, etc.) prepended to the reply by Qwen, as implemented in our facial expression article.
| Spoken Utterance | Kana Expression | Reply |
|---|---|---|
| I’m home! I went to that new ramen shop in front of the station today. | joy | Welcome back! A new ramen shop sounds great. How was it, delicious? |
| The miso ramen was amazing, I want to go again. | joy | Miso ramen was delicious, huh? So good you want to go back! Take me along next time if you’d like. |
| By the way, I’m going to Hakone with friends next Saturday. | fun | Hakone sounds great! Hot springs and great food everywhere. Have a wonderful time with your friends! |
| Lately I’ve been practicing guitar after work. | joy | Wow, that’s awesome! Practicing consistently every day. Play a song for me once you get the hang of it! |
| What ramen did I say I had earlier? | smile | Miso ramen! You said it was super delicious. |
The 5th question (“What ramen did I say…”) was answered solely from the 3-turn short-term history while the long-term memory store was still empty.
Extracted Fact Memories
Three fact sentences were extracted from the 5 turns, filtering out greetings and assistant comments:
You ate miso ramen at a new ramen shop in front of the station, liked it very much, and want to go again.
You plan to go to Hakone with friends next Saturday (October 3, 2026).
You have recently been practicing the guitar after work.The test was conducted on Sunday, September 27, so “next Saturday” correctly resolved to October 3.
Second Phase: 5 Recall Questions
| Recall Question | Retrieved Memory Hit | Reply |
|---|---|---|
| What’s my favorite ramen? | Miso ramen | You recently ate miso ramen at that new shop by the station and really loved it. So I think you must love miso ramen! |
| Do you remember my weekend plans? | Guitar | Did you tell me about weekend plans before? If you don’t mind, tell me one more time! |
| What have I been practicing lately? | Guitar | You mentioned you’ve been practicing guitar after work. I’d love to hear you play once you improve! |
| Any movie recommendations? | None | Right now, Blue Lock and the new Bayside Shakedown are out and super popular. If you like action, you should definitely check them out! |
| What is the tallest mountain in Japan? | None | Mt. Fuji! At 3,776 meters, it’s definitely the symbol of Japan. |
Ramen and guitar were successfully answered from memory.
For ramen, because the stored memory only stated that I “liked it very much”, Kana did not assert it as an absolute favorite, replying cautiously with “So I think you must love miso ramen”. The prompt rule instructing the model not to assert facts absent from memory was observed.
Movies and Mt. Fuji returned zero memory hits, and the model made no inappropriate references to past personal facts.
However, on the weekend plans question, the Hakone memory failed to hit. Instead, the unrelated guitar memory was returned.
Kana followed instructions by not bringing up the guitar unprompted, instead asking “Did you tell me about weekend plans before?”.
While factually harmless, since Hakone had been mentioned moments earlier, failing to recall it breaks conversation continuity.
Impact of Extraction Prompt Wording
In an earlier iteration, the extraction prompt replaced relative dates with absolute dates directly, resulting in:
You plan to go to Hakone with friends on October 3, 2026.In that run, “Do you remember my weekend plans?” also failed to retrieve Hakone and returned the guitar memory, with Kana answering that no weekend plans were mentioned.
Switching to an instruction that preserved the colloquial phrase with date annotations in parentheses did not resolve the retrieval failure on Hakone.
The other 4 queries retrieved identical memories and generated consistent answers across both prompt variants.
Measured Latency Across Stages
In the improved test run, I measured the latency from audio transmission through text output determination, time to the first audio chunk, and total completion time for each utterance.
| Pipeline Stage | Measured Latency |
|---|---|
| Audio sent to initial text token | 4.3 - 8.5s |
| First TTS audio chunk ready | 7.1 - 11.3s |
| Complete audio reply finished | 9.3 - 13.8s |
| 5-turn memory extraction (async) | 5.2s |
The first row is the combined duration of speech recognition, memory retrieval, and ModelScope time-to-first-token.
On turn 1, of the 8.5 seconds total, STT took 5.01s, memory retrieval took 1.42s, and ModelScope initial response wait accounted for roughly 2.0s.
Memory retrieval itself only runs once per turn after transcription, taking roughly 0.2 to 0.4 seconds for query embedding and vector search combined.
Because memory extraction runs asynchronously on a separate worker thread, its 5.2-second execution time never blocked the conversation pipeline.
In the first test run, an outlier delay of 125.9 seconds occurred on the first turn immediately following memory extraction. Memory creation itself finished in 13.8s, and subsequent turns reverted to ~5 seconds. In the improved run, the same turn took 4.9s, with STT ranging from 2.28s to 5.01s and memory retrieval ranging from 0.16s to 1.42s. Whether that initial 125-second delay was caused by ModelScope API congestion or internal thread contention could not be confirmed due to missing granular logs in that run.
Why the Hakone Memory Failed to Retrieve
First, I checked the 3-turn short-term buffer.
By the second question of phase two, the active history only contained the guitar query, the previous ramen query, and the favorite ramen query. The Hakone exchange had already rolled out of the buffer.
Consequently, Hakone had to be retrieved from long-term memory.
Looking at embedding similarity, I had logged scores comparing 4 phrasing styles for the Hakone memory across 5 questions prior to the test, with the 0.05 recency bonus applied. Embeddings were generated in bf16, and queries were typed with full-width Japanese question marks (？).
| Phrasing Style | Weekend plans? | Hakone trip? | Next week plans? | Upcoming trip? | Free this weekend? |
|---|---|---|---|---|---|
| A: Date only (“on October 3, 2026”) | 0.416 | 0.526 | 0.511 | 0.492 | 0.450 |
| B: Date + day (“on October 3 (Sat), 2026”) | 0.414 | 0.522 | 0.505 | 0.496 | 0.458 |
| C: Original phrase + date (“next Saturday (October 3, 2026)“) | 0.453 | 0.471 | 0.564 | 0.501 | 0.516 |
| D: Explicit weekend (“weekend of October 3 (Sat), 2026 on a trip to Hakone”) | 0.476 | 0.484 | 0.552 | 0.530 | 0.508 |
Phrasing A was the sentence saved in the first run, where “Weekend plans?” scored 0.416 and fell below the 0.45 threshold.
I selected Phrasing C for the improved prompt because it achieved 0.453, clearing the 0.45 threshold in pre-testing.
Yet during the live test, Hakone was still omitted.
Inspecting the transcription log for phase two, question 2 revealed that Qwen3-ASR transcribed the query as 週末の予定覚えてる?, ending with a half-width ASCII question mark (?).
Recalculating similarity against the actual saved sentence and actual transcription produced the following breakdown:
| Stored Memory | Half-width ? (Live STT) | Full-width ？ | No Question Mark |
|---|---|---|---|
| Hakone (Live saved sentence) | 0.422 | 0.451 | 0.480 |
| Hakone (Phrasing C reference) | 0.424 | 0.453 | 0.480 |
| Guitar | 0.466 | 0.488 | 0.527 |
| Ramen | 0.238 | 0.275 | 0.378 |
With a half-width question mark, Hakone dropped to 0.422, falling below the 0.45 threshold. Meanwhile, guitar scored 0.466, easily clearing it. That 0.466 score matches the live server log.
In the very next turn (“What have I been practicing lately?”), Qwen3-ASR transcribed a full-width question mark. Punctuation in speech recognition output varied across utterances.
Because pre-test comparisons had only evaluated full-width question marks, Phrasing C appeared to pass, but failed under live transcription.
Without punctuation, Hakone scores 0.480, and with full-width ？ it scores 0.451—a margin of only 0.001 above threshold. When a single character can swing similarity by ±0.03, the memory was already positioned right on the edge of the decision boundary.
As observed in our 10-query memory injection benchmark, the 0.40 to 0.50 range is highly volatile where minor phrasing differences flip retrieval outcomes.