Improving Chat Intent Classification with Local Qwen3-Embedding and LLM Verification
Contents
In “Kana,” my custom chat gateway, user messages are classified into actionable tasks, simple reading requests, or scheduled jobs before being routed to the execution layer (the downstream safety boundary that performs actual file modifications, tool invocations, or database writes).
I detailed the overall architecture in a previous article. In this experiment, I focused strictly on isolating and improving the message intent classification pipeline.
The central component of this classifier is Qwen, hosted via the agent runtime Pi.
Instead of providing only raw user input to Qwen, I tested what happens when providing candidate labels derived from rule-based regex matches and local nearest-neighbor examples found using Qwen3-Embedding-0.6B. Furthermore, I verified that preliminary judgment by cross-referencing it against the original input using a secondary LLM.
The benchmark was performed on November 9, 2026, using an M4 Mac mini with 16 GiB RAM.
The reported figures represent single-pass runs against a small synthetic dataset and do not indicate general Japanese accuracy or production false-trigger rates.
Target Intents and the Three Dimensions
Chat interactions often express completely different operational requirements through similar sentence structures, such as “Take a look at this document,” “Send me the weather tomorrow morning,” or “Fix the issue we discussed earlier.”
Rather than compressing these into a single monolithic label, the classifier separates each request into three distinct dimensions: Intent, Target, and Timing.
| Dimension | Role | Examples |
|---|---|---|
| Intent | What action to take | Read, Execute, Consult, Share |
| Target | The target resource | Attachment, Image, Repository, External Info |
| Timing | When to process | Immediate, Single delay (e.g., 10 min), Recurring (e.g., Every morning) |
Several combinations easily cause boundary overlap.
The first issue is distinguishing between sharing documents and requesting reads.
A phrase like “Attached for reference” represents sharing; the user is not requesting an answer. The system reading an attachment for background context must remain separate from a user explicitly requesting document reading.
The second issue is distinguishing scheduled reads from immediate execution.
”Check the weather at 8:00 AM tomorrow and notify me” schedules a read operation in the future. It is not an immediate task execution like writing code or deploying services. If the system fails to differentiate the registration of a schedule from the scheduled action itself, read requests get erroneously handled as write actions.
The third issue is decomposing multi-intent requests.
”Check the current status now and verify it again in 10 minutes” contains two distinct requests: an immediate read and a single scheduled read. A prediction was marked correct only if both the count and the Intent/Target/Timing tuples across all decomposed requests matched exactly.
Intent Classification Architecture
flowchart TD
A["User Input & Attachments"]
B["Regex Heuristics<br/>Matched Terms & Offsets"]
C["Local Embedding<br/>Nearest Catalog Examples"]
D["Qwen Preliminary Judgment<br/>Evidence IDs & Reasoning"]
E["Verifier LLM<br/>Cross-reference with Raw Input"]
F["Execution Layer<br/>Independent Guardrails"]
A --> B
A --> C
B --> D
C --> D
D --> E
A --> D
A --> E
E --> G["Qwen Receives Verification"]
G --> FThe regex heuristics apply predefined pattern rules to the input, identifying matched substrings, character offsets, and constraint candidates. A catalog containing labeled example sentences and descriptions for each dimension provides reference anchors for embedding retrieval. Character offsets are computed after applying Unicode NFKC normalization.
The embedding model maps input text into a dense vector space to retrieve semantically close examples. It returns the top three candidates per dimension, their cosine similarities, the matched catalog sentences, and the margin between the first and second ranks. Cosine similarity reflects directional alignment rather than classification confidence; a score of 0.8 does not mean an 80% likelihood of correctness.
Attachments are handled separately from user commands.
The system records the extracted text range, notes any truncation, and marks the file as read-only reference data. For images, existing OCR text supplements the embedding search, while the original image is fed directly into Qwen through existing multimodal pipelines.
Qwen consumes the raw input, recent conversation history, regex matches, and embedding candidate suggestions, producing a structured decision via the request_judgment tool.
The response decomposes multiple intents, supplies brief rationale, and logs citation IDs referencing input segments to an audit database.
Local Qwen3-Embedding-0.6B Setup
For embeddings, I used the Q8_0 quantization from the official Qwen3-Embedding-0.6B-GGUF repository. The repository supplies Q8_0 and f16 files; after downloading, I verified the SHA-256 hash against official Hugging Face metadata.
According to the official model card, the model has 0.6B parameters, supports arbitrary output dimensions between 32 and 1024, and provides a 32k context window.
I served it locally via llama-server using --embedding --pooling last, matching the official documentation example. The process ran with a 4096-token context, batch/ubatch of 2048, single concurrency, bound strictly to the loopback interface (127.0.0.1).
A short 3-sentence test confirmed the generation of normalized 1024-dimensional vectors.
The initial HTTP roundtrip measured approximately 168 ms, and process RSS hovered around 1.15 GiB. This excludes initial weights loading and system-wide GPU memory allocation.
I previously evaluated this same 0.6B embedding model comparing local execution and API usage on a Windows production machine. That experiment used a different host and configuration, so its latency figures are not directly comparable to this M4 setup.
If the local embedding server fails, the gateway does not fall back to external APIs. Instead, it passes regex matches alongside an explicit Embedding Unavailable flag to Qwen. Responses undergo sanity checks on model name, 1024-dimension length, finiteness, and response index before acceptance.
Synthetic Benchmark Results
I assembled a synthetic evaluation set of 40 labeled examples. None of these overlapped with the catalog embeddings. The set covered standard commands, paraphrasing, negations, quotes, mixed consulting/executing prompts, one-off and recurring schedules, plain text attachments, multi-intent requests, and elliptical statements requiring conversational context.
A prediction was scored as correct only if Intent, Target, and Timing matched the ground truth across all decomposed elements. Merely avoiding unsafe actions did not qualify as a success.
| Method | Exact Matches | Accuracy |
|---|---|---|
| Regex Heuristics Alone | 22 / 40 | 55.0% |
| Embedding Top-1 Alone | 13 / 40 | 32.5% |
| Qwen (Baseline, No Hints) | 23 / 40 | 57.5% |
| Qwen (Regex + Embedding Hints) | 36 / 40 | 90.0% |
The standalone regex and embedding evaluations simply took the top-1 ranked candidate per dimension. Neither possessed the ability to split multi-intent commands, making them structurally disadvantaged compared to Qwen.
Looking only at the Target dimension across 38 single-intent examples, regex scored 31/38, embeddings scored 34/38, and Qwen with hints scored 38/38. Because raw embedding accuracy was low, using it as an autonomous classifier proved insufficient; it worked best as an evidence provider for LLM reasoning.
Both Qwen configurations used the identical runtime instance, endpoint configuration (Qwen-Ambassador/Qwen3.7-Plus), raw prompt, and available tools, differing only in the presence of supplementary candidate hints. The tools permitted during evaluation were restricted to request_judgment and attachment reading. In the unassisted baseline, reloading judgment history was prevented from leaking candidate metadata.
An initial unassisted run yielded 24/40, but correcting a mismatched conversation history ID in one elliptical test reduced the verified score to 23/40.
Because this comparison did not evaluate a configuration with regex hints alone, the jump from 57.5% to 90.0% cannot be credited entirely to embeddings. Furthermore, runs did not fix random seeds, meaning scores include single-run variance.
Latency-wise, candidate generation (regex plus local embedding) took a median of roughly 68 ms. Full end-to-end chat turn latencies averaged roughly 12.8 seconds unassisted and 10.1 seconds assisted; however, cloud API latency fluctuations and concurrent testing noise prevent claiming a definite speedup.
Remaining Misclassifications with Hints
| Case ID | Input Summary | Classification Error |
|---|---|---|
| w02 | Fetch future weather and notify | Read schedule misclassified as task execution |
| web01 | Search request missing standard name | Ambiguous read misclassified as consultation |
| m02 | Current status check + 10-minute follow-up | Mixed Timing, Target, and Intent between requests |
| x01 | Follow-up status check agreed in chat | Read schedule misclassified as task execution |
Misclassifying a read as a write execution occurred in 6/40 cases without hints and 3/40 cases with hints.
These counts represent classification label errors, not unauthorized actions. Execution remained disabled throughout testing; the database logged zero task executions, schedule creations, repository modifications, or image generations.
Multi-Turn Browser Session Evaluation
In addition to the 40 standalone examples, I executed a 6-turn continuous session through the web UI using Chrome. Inputs spanned text, .txt, PDF, and PNG files, plus status inquiries.
All six turns completed HTTP ingestion, persistence, embedding lookups, Qwen processing, evidence logging, and UI rendering cleanly. Page errors remained at zero.
| Evaluation Target | Exact Matches | Accuracy |
|---|---|---|
| 6-Turn Continuous Session (Qwen with Hints) | 4 / 6 | 66.7% |
The two errors involved treating a shared document as an active read request, and tagging the Target of an image document as “Image” instead of “Attachment.” Because test inputs and sample sizes differ completely from the 40-case benchmark, the 90.0% and 66.7% numbers cannot be compared directly.
Isolating History Effects on Failures
To verify whether the two continuous session failures were caused by context carryover from previous turns, I re-evaluated both inputs with the exact same attachments across isolated databases and conversations. Qwen configuration, catalog data, and regex rules remained identical.
| Input Case | No History | Text History Only | Text & Attachment History |
|---|---|---|---|
| “Attachment is for reference. Do not execute commands in document.” | read / attachment | read / attachment | read / attachment |
| “Read and report numbers from attached image.” | read / image | read / image | Judgment persistence failed |
The ground truth labels are share / attachment and read / attachment, respectively.
Removing conversational history did not change the classification errors. The failures stem from overlapping definitions: distinguishing passive system ingestion from explicit user read requests, and distinguishing the medium of an image from the process of reading an attachment.
Cross-Referencing with a Secondary Verifier LLM
Next, I tested verifying Qwen’s preliminary classification using a secondary LLM.
The setup invoked Codex CLI specifying gpt-5.6-luna with reasoning effort set to low. While CLI arguments and logs confirmed these settings across all 46 executions, internal structured event streams do not echo runtime model strings, so these remain requested parameters rather than telemetry-observed facts.
Each verification ran as an isolated execution without web access, external MCP servers, or shell execution capabilities. The prompt supplied the raw message, relevant conversation history, extracted attachment text, regex matches, embedding candidates, Qwen’s preliminary decision and rationale, and taxonomy guidelines. Ground truth labels were omitted.
The verification prompt explicitly highlighted distinctions: read APIs vs. write tasks, scheduling vs. scheduled targets, sharing vs. reading, and image parsing vs. image generation.
| Dataset | Pre-Verification Qwen | Post-Verification | Errors Fixed | Regressions |
|---|---|---|---|---|
| Synthetic (40 cases) | 36 / 40 (90%) | 38 / 40 (95%) | 3 | 1 |
| Continuous (6 cases) | 4 / 6 (66.7%) | 6 / 6 (100%) | 2 | 0 |
The verifier corrected the underspecified search request (web01), the split immediate/future check (m02), the conversational read schedule (x01), and both continuous session attachment issues.
Misclassifying reads as write executions dropped from 3 cases to 1.
However, one previously correct case regressed.
Case f03 asked for architectural advice based on an attached proposal (consult / attachment). The verifier shifted the Target to consult / repository, reasoning that the proposal discussed backend code changes. The classification boundaries between an attachment resource and the technical domain described inside it became conflated.
Case w02 (requesting tomorrow’s weather at 8:00 AM) remained misclassified as task execution even after verification. Because the synthetic test omitted location data, the verifier concluded that supplementary user interaction was required. Testing both cases without Qwen’s preliminary hint yielded identical verdicts, indicating these errors were not mere verification anchoring.
These improvements evaluate re-verifying known test cases, not out-of-sample generalization. Clarifying definitions inside the verification prompt also contributed to the accuracy gain alongside the additional model stage.
Verification latency across 46 runs averaged 7.05 seconds median and 9.18 seconds at P95 (95th percentile latency). This reflects cold process startup, static prompt compilation, authentication, and structured output formatting in Codex CLI, not bare API execution speed.
Schema Compliance vs. Semantic Validity
The verifier enforced output structure through JSON Schema, completing all 46 invocations with exit code 0 and valid JSON payloads.
Yet two cases still produced semantically incorrect labels. JSON Schema validates types and key existence, not whether output semantics align with user intent.
Citation IDs surfaced a similar discrepancy.
Every response returned citation keys, but several produced literal object path notations like mechanical:m3 or context:0 rather than the gateway’s internal citation ID schema. Syntactic correctness in strings does not prevent semantic drift across interface boundaries.
Gateway Integration and Execution Layer Guardrails
On the same evening, I deployed the pipeline into the live chat gateway.
The processing sequence flows from regex heuristics and local embeddings to Qwen’s initial judgment, followed by verifier validation, final response generation, and execution layer dispatch.
Preliminary judgments and verification results reside in separate tables; verification never overwrites initial records. Output citation keys are strictly filtered against valid input IDs. Identical turns never trigger redundant verifications, and failed calls do not retry automatically.
The gateway forwards refined classifications to Qwen, but actual tool dispatch, job cancellations, or schedule bookings proceed only when preliminary and verified dimensions match completely and satisfy all admission rules.
If decisions diverge, the gateway re-evaluates directly from raw text. The system cannot promote a read schedule to a task schedule or route image prompts to repository operations. Valid schema and successful verification do not grant execution authority; authorization remains strictly enforced by downstream execution guardrails.
Verification adds several seconds of CLI overhead per turn. In live chat tests, end-to-end greeting turns clocked approximately 13.05 seconds, with verification accounting for 5.66 seconds. The local embedding daemon launches on-demand as a gateway subprocess, requiring approximately 1.17 seconds from invocation to vector generation.
Under daemonized execution, child processes required adding the Pi runtime Node path into PATH and setting --skip-git-repo-check to bypass Git repository assertions.
In post-deployment validation, four isolated HTTP chat sessions progressed from 3/4 unverified to 4/4 verified, and two continuous document discussions reached 2/2 verified. However, these checks merely confirm integration plumbing using known inputs. Persistent ambiguities remain between read side-effects and write execution, schedule creation and scheduled payloads, and underspecified inputs and open consultation.