Why Qwen3.8-Max on ModelScope Couldn't Tell Me Today's Weather, and How Tool Calling with ddgs, JMA Forecasts, and RSS Fixed It
Contents
At the end of the post where I connected StackChan to a voice chat server, I asked “What’s the weather like now?” and was told it couldn’t see outside.
At that point, the voice chat server only called Qwen through ModelScope’s API and had no way to look anything up.
Just after midnight on October 20, I asked the same server “What’s the weather in Tokyo today?” from the Xperia tablet I turned into a photo frame, and again got “Sorry, I don’t know today’s weather in Tokyo.”
Yet this server calls Qwen with enable_search: True, which turns on ModelScope’s built-in web search. I wanted to know why it still couldn’t tell me the weather with search turned on.
I asked the Claude instance running on the voice chat server’s PC to do the investigation and make the changes. The numbers in this post come from its reports and from the output of the test scripts it sent along with them. All conversations were in Japanese, and the quotes below are translations.
Voice chat server setup
| Part | Details |
|---|---|
| Transcription | Qwen3-ASR-0.6B (on the server’s GPU) |
| LLM that writes replies | Qwen3.8-Max (via ModelScope API-Inference) |
| Speech output | A separate TTS server (turns text into speech) |
| Who talks to it | StackChan and the Xperia tablet |
Replies are in the spoken style of a character named Kana, and are kept to two sentences because they are read aloud.
The first instructions given to the LLM (the system prompt) contain the current date and time, the CO₂, temperature, and humidity measured by StackChan, and memories retrieved from long-term memory.
Search was running
First, I checked whether the built-in search was actually running.
Using a short system prompt that contained only the date and time, I sent the same questions with and without enable_search and compared the input token counts in the API responses. The input token count shows how long the text passed to the LLM was.
| Question | Input tokens with search | Input tokens without search |
|---|---|---|
| What’s the weather in Tokyo today? | 3,427 | 49 |
| What’s in the news today? | 4,639 | 47 |
| What’s the dollar-yen rate right now? | 4,663 | 48 |
With search on, the input grew by more than 3,000 tokens, so search results were being added to the prompt.
Even so, the weather reply said the results “do not include real-time weather data for Tokyo on October 20, 2026 (today).”
Without a system prompt, it answered “According to the knowledge base (知识库), today’s (October 16, 2026) weather in Tokyo is…” with the weather for a date four days earlier. Asked about dollar-yen, it gave a rate from around April 2022.
In other words, whatever the search fetched did not include today’s Tokyo weather, so Qwen said it didn’t know.
The character settings were not the cause
The system prompt includes the line “Answer concretely about what is written, and say you don’t know about what is not written.” I wondered whether that line was stopping it from using the search results.
I sent the 00:03 question four times each with three prompts: the actual prompt, the prompt without that line, and the prompt without the memory section.
| Prompt | Times it answered “I don’t know” |
|---|---|
| Actual prompt | 4 out of 4 |
| Without the “say you don’t know” line | 4 out of 4 |
| Without the memory section | 4 out of 4 |
All 12 answers were “I don’t know.” Changing the prompt did not change the reply.
Where the built-in search gets its information
Next, I asked it to list every source URL and site name it got the information from.
The source in the reply was “中国天气网 (China Weather Network, data from the Central Meteorological Observatory),” and for the URL it said “no URL is included in the provided text.” Since the LLM wrote that source itself, there is no way to confirm it actually came from there.
The reply also mixed in Tokyo weather for different dates, from October 16 to 19.
Alibaba’s DashScope has a search_options parameter that makes the search return its sources. I tested whether it works through ModelScope.
| Option | Result |
|---|---|
Return sources (enable_source) | No new fields in the response, and the same input token count as without it |
Add citations (enable_citation) | Same as above |
| A nonexistent option | Same as above. No error either |
Search strategy agent | 400 error. Not available for this model |
Search strategy max | Both the input token count and the answer changed |
Since agent returned an error and max changed the result, the options were reaching Alibaba’s side.
Still, no sources came back, and there was no option to choose the search target or region.
In the same time window, asking with the date-only prompt sometimes produced “light rain, high of 24–26°C” as the weather for October 20.
The Japan Meteorological Agency (JMA) forecast for Tokyo on the 20th was “cloudy, rain in some places until evening” with a high of 27°C, so that did not match. With the max search strategy, it even returned a forecast of “high and low both 0°C, very cold.”
With the built-in search, today’s weather was sometimes missing. Even when it was there, it did not match the JMA forecast, and no sources came back. The dollar-yen rate was old, and the news had Chinese translations mixed in.
I dropped the built-in search and decided to provide search on the server side.
Giving Qwen tools
With function calling, you give the LLM the names and descriptions of tools. When the LLM decides it needs one, it returns a reply that says “call this tool with these arguments.”
The LLM does not search by itself. The search is done by Python code on the server, and when the results are sent back to the LLM, it uses them to write the reply.
To see whether this works with Qwen3.8-Max on ModelScope, I gave it a single tool called web_search and tried six questions.
| Question | Called the tool? | Search query |
|---|---|---|
| What’s in the news today? | Yes | October 20, 2026 news |
| What’s the dollar-yen rate right now? | Yes | dollar yen exchange rate now |
| How did the Japan Series go yesterday? | Yes | Japan Series October 19, 2026 result |
| I’m getting sleepy. | No | - |
| How tall is Mt. Fuji? | No | - |
| I’m heading over to Suzuki-san’s place. | No | - |
It did not search for small talk or things it already knew, and called the tool only for the three questions that needed new information. The search queries included the date, too.
Choosing a search service
Before deciding what to search with, I looked at which search APIs are available.
Google’s Custom Search JSON API is closed to new customers, and existing customers lose access on January 1, 2027. Microsoft’s Bing Search API was retired on August 11, 2025.
So I started with ddgs, which costs nothing extra.
ddgs is a Python library that collects results from several search services such as DuckDuckGo, and this server uses it without configuring any API key. I decided to swap in Brave’s search API if problems came up.
Answering with ddgs search results
I fetched five Japan/Japanese-language results and passed their titles, snippets, and URLs straight back to the LLM.
| Question | Reply | Total time |
|---|---|---|
| What’s in the news today? | Couldn’t really tell what today’s news was about | 6.7 s |
| What’s the dollar-yen rate right now? | About 158 yen 40 sen to the dollar | 4.7 s |
| How did the Japan Series go yesterday? | Answered “I don’t know” without searching | 1.2 s |
For dollar-yen, it used the “158.41-158.43 (10/19 23:59)” figure from a Nikkei snippet in the search results.
For the news, the results were mostly NHK program pages and news site index pages, with no actual stories.
For the Japan Series, it answered without calling the tool.
After I added “For things outside your knowledge, such as recent events, news, markets, and game results, don’t guess. Look them up with web_search before answering” to the system prompt, it started searching.
The reply was “This year’s Japan Series opens on October 24, so no games had started yet as of yesterday!”, which matched the pro baseball data page in the search results.
For the news, I made it also use ddgs’s news search.
This time it said today was the day the drama “Midnight Diner 6,” starring Kaoru Kobayashi, starts airing for the first time in seven years. The top search result was an article announcing the drama’s premiere, which is off the mark as today’s news.
For the weather, it made things up instead of calling the tool
For the weather, I decided to use the forecast JSON that the JMA forecast pages load. It contains forecasts for today, tomorrow, and the day after, plus a one-week forecast, for each prefecture.
I created a tool called weather that fetches the forecast for a given prefecture.
But when I asked “What’s the weather in Tokyo today.” (ending with a period), it answered without calling the tool: “It’s sunny in Tokyo today, with a forecast high of 22 degrees. The chance of rain is 0 perce—”. Asking again gave the same result.
The forecast for Tokyo on the 20th was “cloudy, rain in some places until evening,” so this was made up.
Keeping the same conditions and changing only the wording gave this:
| Question (translated) | Called the tool? |
|---|---|
| What’s the weather in Tokyo today. (今日の東京の天気は。, ending with 。) | No |
| What’s the weather in Tokyo today? | Yes |
| Tell me Tokyo’s weather today | Yes |
| What’s the weather in Osaka today. | Yes |
| Will it rain in Osaka tomorrow? | Yes |
Only “What’s the weather in Tokyo today.” ending with a Japanese period (。) failed to call the tool. This test did not establish whether the period itself was the cause.
To rule out ModelScope quietly searching anyway, I also compared it with enable_search: False set explicitly. The input token count was 1,176 in both cases, so no search that adds results to the prompt was happening.
Leaving the decision to call the tool entirely to the LLM failed on the question I ask most often.
I decided to always put the Tokyo forecast JSON into the system prompt, and kept fetching other prefectures with weather.
JMA updates its forecasts at 5:00, 11:00, and 17:00, so a fetched forecast is reused for 30 minutes instead of being fetched again.
The second request returned an empty reply
When I asked about dollar-yen, the reply sometimes ended up empty.
Qwen called search in the first request, and when the results were sent in a second request, it tried to call search again.
The server did not handle a search call in the second round, so it ended without any output.
I made searches repeatable, with up to two searches per reply. If there is still no answer, it replies “Sorry, I couldn’t look that up properly.”
Searches are capped at 100 per day, and the search query, the fetched URLs, and errors are written to search.log. When a reply looks wrong, I can go back and check what it was based on.
Asking again from the tablet
At 1:21, I asked the same “What’s the weather in Tokyo today.” from the tablet, and this time it answered.
Tokyo is cloudy today, with rain in some places tonight. Temperatures will be around 20 to 27 degrees, and since rain is possible until evening, an umbrella would be a good idea.
The JMA Tokyo forecast JSON fetched at that time was the one issued at 17:00 on the 19th. The content at this URL is overwritten each time a new forecast is issued.
Compared with it, the 20th’s “cloudy, rain in some places until evening” and the 20°C and 27°C temperatures matched the reply, but “rain in some places tonight” was the text for the 19th.
The forecast JSON contains forecasts from the time it was issued onward.
At 1 a.m., the next forecast at 5:00 had not come out yet, so the 19th’s entry was still at the top, and the text for the night of the 19th got mixed with the 20th’s.
First, I wrote into the prompt that it should read which values go with which dates and times, and should not use values for days before today.
With that, “What’s the weather in Tokyo today” and “Do I need an umbrella today?” were correct all four times, but “Will it rain tonight?” was answered with the text for the night of the 19th all four times.
Next, I made the server remove forecasts whose period had ended before sending them to the LLM.
Forecasts by time slot, such as weather and chance of rain, are removed once the next time slot begins. Temperatures are point values, the morning low and the daytime high, so they are removed only after their date has passed.
| ”Will it rain tonight?” around 1 a.m. | Forecast it answered with |
|---|---|
| Before removing | The forecast for the night of the 19th, all 4 times |
| After removing | The forecast for the 20th, all 4 times |
Some days in the weekly forecast have no text weather, only a numeric weather code.
I added to the prompt an explanation that the hundreds digit of the code is treated as the main weather (1 is sunny, 2 is cloudy, 3 is rain, 4 is snow), along with the dates and days of the week for the coming week starting today.
News that stopped at “I’ll tell you”
At 1:40, I asked from the tablet, “Get me the latest news list from Yahoo News.” In the transcript, the word for “list” (一覧) had become a similar-sounding word (一見).
The whole reply was one sentence: “I can’t get the latest Yahoo News articles, so I’ll search for the latest news and tell you instead.”
Qwen had said “I’ll search and tell you” while declining to open the site, then ended the reply without calling search.
When I sent the same question several times, sometimes it searched and gave two news topics, and sometimes it declined with “I don’t have a way to get the latest article list, so I don’t know.”
Getting headlines from RSS
RSS is data that news sites and others publish in a fixed format, listing article headlines and URLs.
I stopped searching for news lists and created a news tool that fetches RSS from Yahoo! News Japan and NHK.
| Source | Categories | Content |
|---|---|---|
| Yahoo! News Japan | Top, domestic, world, business, entertainment, sports, IT, science, local | Headlines |
| NHK | Top, society, lifestyle, science and culture, politics, business, world, sports | Headlines and summaries. Up to 15, newest first |
RSS came back in 0.3–0.7 seconds, and a news reply took about 4 seconds in total.
When someone asks about a specific topic in detail, it still uses web_search as before.
Answering with old news without calling the tool
Even after adding RSS, when I asked “Any news?”, it answered without calling the tool two out of three times.
It talked about past news such as the 2021 Nobel Prize in Physics and presented it as today’s news. “What’s in the news today?” also sometimes got past news without a tool call.
So when the transcribed text contains the word “news” (ニュース), the tool call is required from the first request. Adding tool_choice="required" to the API call makes the LLM always return a tool call.
For other questions, if the last sentence of the reply is an announcement such as “I’ll look it up” or “I’ll search and tell you” and no tool was called, the server asks again with the tool call required.
Offers like “I can look that up for you” are treated separately from such announcements.
News replies would be long if everything were read aloud, so I limited them to two headlines and about 70 Japanese characters in total.
| Question | Reply (time to first sentence) |
|---|---|
| Any news? | 2 headlines from Yahoo! News top stories (4.2 s) |
| Any sports news? | 2 headlines from Yahoo! News sports (3.6 s) |
| Tell me the NHK news | 2 headlines from NHK top stories (3.8 s) |
| Get me the latest news list from Yahoo News. | 2 headlines from Yahoo! News top stories (3.8 s) |
Replies after the rebuild
| Question | Result |
|---|---|
| Tokyo weather today, umbrella, the weekend, Osaka tomorrow | Matched the JMA forecast |
| How did the Japan Series go yesterday? | No game. Opens October 24 with DeNA vs. SoftBank |
| News list | 2 headlines from RSS |
| What’s the dollar-yen rate right now? | Gave a figure (low 158 yen range) only 1 out of 3 times |
| Height of Mt. Fuji, “I’m getting sleepy” | Answered without searching |
Time to the first sentence was 5–8 seconds for replies that searched and 1.5–2 seconds for replies that did not. Because the Tokyo weather was already in the prompt, it answered that without searching.
For dollar-yen, the search snippets often did not contain the rate.
With ddgs, either the news search or the regular search occasionally returned a “no results” error, and in those cases it answered from the other one’s results.