How I designed long-term StackChan memory with Qwen3-Embedding and Qdrant

I already have a Qwen3-ASR and Qwen voice-chat server, and a CoreS3 can call it asynchronously, so the next addition is memory that survives across sessions.

Right now, each request carries conversation history forward, but ending the session removes it.
Sending the entire log to Qwen would make the input grow with every conversation, so I want Qwen to search old conversations only when I ask about something like “the Fujitsu PC I bought before” or “the setup we decided on last time.”

Rather than move everything to StackChanWorld’s conversation API, I am adding SQLite, Qwen3-Embedding-0.6B, and Qdrant to the backend that already works.

Source turns stay in SQLite

I first had to decide how much conversation belongs in one vector.
One vector per turn would leave many entries such as “that,” “the other one,” and “right” with little meaning on their own.
A whole-session vector has the opposite problem once the conversation changes topics: it mixes them and makes either one harder to retrieve.

I will group a few exchanges on one topic into a searchable memory, keep every original turn in SQLite, and put only the summary vector and retrieval fields in Qdrant.

flowchart TD
    A[Original turns] --> B[SQLite]
    B --> C[Summarize by topic]
    C --> D[Qwen3-Embedding]
    D --> E[Qdrant]
    E --> F[Return to source turns<br/>through memory_id]

Searchable-memory fields

{
  "memory_id": "mem_01...",
  "topic": "StackChan voice backend",
  "summary": "An RTX 3050 Ti laptop runs Qwen3-ASR on the CPU and Irodori-TTS on the GPU, with voice round trips working from a CoreS3.",
  "entities": ["StackChan", "CoreS3", "RTX 3050 Ti", "Qwen3-ASR", "Irodori-TTS"],
  "memory_type": "episode",
  "occurred_at": "2026-07-23T01:00:00+09:00",
  "source_turn_ids": [381, 382, 383],
  "status": "active"
}

If Qdrant’s summary search finds several candidates, or if the answer depends on a number or date, source_turn_ids points back to the SQLite turns.
If a Qdrant write fails, the source log remains in SQLite.
A pending state is enough to retry indexing, and the same records can rebuild the index if the Qdrant data is removed.

The 0.6B model on the CPU

The voice-chat server used 3,945 of the GPU’s 4,096 MiB with Irodori-TTS kept resident.
That leaves no room to add Qwen3-Embedding on the GPU, so the 0.6B model will run on the CPU.

The official Qwen3 Embedding repository lists 0.6B, 4B, and 8B models.
The 0.6B model produces up to 1,024 dimensions and supports Matryoshka Representation Learning (MRL) for shorter output dimensions.
A 1,024-dimensional float32 vector occupies 4,096 bytes before metadata.
I will measure RAM use and latency beside Qwen3-ASR before worrying about vector storage.
Those results will decide whether the existing FastAPI process loads the model or a separate process does.

Qdrant local mode for v1

Qdrant stores JSON payloads alongside vectors.
Its filtering documentation supports AND, OR, and NOT clauses, so I can narrow a search by entity, date, memory type, and status.
The Python client also has a local mode that needs no Qdrant server.
QdrantClient(path="path/to/db") persists the local database to disk, which should be enough for the first version.

When Qwen searches memory

A string match on Japanese phrases such as 前に (“before”), この前 (“last time”), or 覚えてる (“remember?”) is easy to write, but it would also fire on 前に進んで (“move forward”).
It would miss “What did I run on the RTX 3050 Ti?” because none of those trigger words appear.

I considered calling a separate classifier on every turn.
That would add another Qwen request even to small talk that needs no memory.

I have already tested function calling through ModelScope’s Qwen API.
In that test, the application received tool_calls, ran a local function, and returned its result to Qwen so response generation could continue.
Long-term memory will use the same path with two tools, search_memory and read_memory_source.
A turn that needs no memory can finish with the first Qwen call.

flowchart TD
    A[Transcribe with Qwen3-ASR] --> B[Send fixed name, speaking style, and recent turns to Qwen]
    B --> C{Call search_memory?}
    C -->|No| D[Reply]
    C -->|Yes| E[Embed the search query with Qwen3-Embedding]
    E --> F[Retrieve candidates from Qdrant]
    F --> G{Need source turns?}
    G -->|Yes| H[Read them from SQLite]
    G -->|No| I[Return search results to Qwen]
    H --> I
    I --> D
    D --> J[Irodori-TTS]
    J --> K[Play on CoreS3]

The application puts Qdrant similarity in retrieval_score and uses it only for ranking.
A value of 0.91 does not mean “91% confidence” that the memory is correct.
After reading the original turns, the application sets a separate boolean, source_checked.

A fixed formula such as 0.6 × similarity + 0.25 × entity match + 0.15 × recency would favor newer memories even when I ask for an old conversation.
An explicit date range takes priority.
Without one, recency can break ties, while named entities and memory type narrow the candidates through Qdrant filters.

Searchable-memory creation in the background

For a turn that needs an old memory, Qwen receives the Qdrant candidates before replying.
After writing the raw turns to SQLite, the server generates the reply audio and builds the searchable memory in parallel.

flowchart TD
    A[Write turns to SQLite] --> B[Continue the reply]
    A --> C[Background queue]
    C --> D[Segment topics and summarize]
    D --> E[Embed]
    E --> F[Write to Qdrant]

If the PC shuts down before the queue finishes, the next startup can enqueue the remaining pending records.
Greetings, filler, and repeated wording stay only in the source log.
Events, decisions, and preferences likely to matter later become searchable memories.
After implementation, I will count how often filler is excluded and events or preferences are retained in actual conversation logs.

Persona configuration stays separate from memory

Layer	Contents	Handling
Persona	Name, first-person pronoun, speaking style, answer length	Retained until configuration changes
Session	Recent turns and current topic	Retained only during the conversation
Memory	Past events, preferences, and decisions	Retained across sessions
Body	Expressions, lip sync, and servo actions	Executed by CoreS3

The persona remains in a separate configuration file, and retrieved conversations supply context only for Qwen’s reply.

The CoreS3 response contains three fields: reply text, an expression ID, and an action ID.
happy selects an existing smile image and nod selects a predefined servo action.
Qwen returns only the IDs; the CoreS3-side mapping manages servo angles.

{
  "reply": "That's the one we tried on the RTX 3050 Ti laptop.",
  "emotion": "happy",
  "action": "nod"
}

Storing Qwen’s internal states

I also considered keeping Qwen’s attention values or hidden states instead of using one embedding for each topic-group summary.
The ModelScope API used by this server does not expose those internal values.

Even with a local model, the Transformers model-output specification gives attention tensors for every layer with shape batch × heads × sequence × sequence, and hidden states with shape batch × sequence × hidden size.
Both depend on the model and token count rather than producing one fixed-length vector per conversation.

The Transformers KV-cache documentation describes how it reuses key and value calculations from generated tokens to reduce repeated work during generation.
It cannot retrieve the old conversation where I discussed a Fujitsu PC.

Offline scoring after each test conversation

An earlier design used a second Qwen call to score every reply.
The extra call would delay TTS even for an ordinary one-line response.
I removed scoring from the live path.
A separate Qwen call will score each completed 10-turn and 50-turn test conversation.
The tests count false search triggers and missed searches.
They also check whether Qwen asks for clarification when several memories match and favors the new memory after I correct an earlier statement.