Skip to content

fix(chat): support tool_choice in Qwen3.5-VL chat handler#108

Closed
abdullah-cod9 wants to merge 1 commit intoJamePeng:mainfrom
abdullah-cod9:main
Closed

fix(chat): support tool_choice in Qwen3.5-VL chat handler#108
abdullah-cod9 wants to merge 1 commit intoJamePeng:mainfrom
abdullah-cod9:main

Conversation

@abdullah-cod9
Copy link
Copy Markdown

Problem

tool_choice parameter is ignored in Qwen35ChatHandler.
Tools are always injected into the prompt regardless of the value passed,
because the chat template never references tool_choice.

Root Cause

The CHAT_FORMAT string in Qwen35ChatHandler only checks:

{%- if tools and tools is iterable -%}

with no condition on tool_choice.

Solution

Added tool_choice handling in the template:

Value Behavior
"none" Tools not injected at all
"required" Model instructed to always call a tool
"auto" / None Model follows each tool's description

Testing

  • tool_choice="none" → no tools in prompt
  • tool_choice="required" → model calls a tool
  • tool_choice="auto" → model decides based on tool description

Closes #107

- 'none': tools are not injected into the prompt
- 'required': model is instructed to always call a tool
- 'auto'/None: model follows each tool's description to decide

Fixes: tools were always injected regardless of tool_choice value
@JamePeng
Copy link
Copy Markdown
Owner

JamePeng commented Apr 6, 2026

The original Qwen 3.5 chat template didn't have any logic regarding tool_choice.

https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/chat_template.jinja

@abdullah-cod9
Copy link
Copy Markdown
Author

The original Qwen 3.5 chat template didn't have any logic regarding tool_choice.

https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/chat_template.jinja

I didn't notice that, thank you for pointing it out!

Is tool_choice only supported if the original model's chat template explicitly handles it?

If the original chat template doesn't support it, what would be the downside of adding the logic directly in llama-cpp-python's version of the template?

I'm asking to better understand the design philosophy here — is there a reason to stay strictly in sync with the original template?

@JamePeng
Copy link
Copy Markdown
Owner

JamePeng commented Apr 6, 2026

Current adaptations primarily involve injecting the image_url into the multimodal component and passing in Jinja variables. However, it is strongly recommended to retain the original chat template. This may have a smaller impact on models with large parameters, but could potentially affect models with small parameters.

Of course, you could also share your code examples and why this variable is so important, so I can see if there are any other potential issues.

@abdullah-cod9
Copy link
Copy Markdown
Author

Thank you for the explanation!

My use case: I have a set of tools with a button in the UI to enable or disable them via tool_choice. Since the README doesn't mention that tool_choice isn't supported on all models, I assumed it was a bug when it didn't work with Qwen3.5.

I tested the change on Qwen3.5 2B and it worked fine without any noticeable impact on the model's behavior.

That said, I understand your concern about staying close to the original template, especially for smaller models. If you'd prefer to keep the template unchanged, feel free to close the PR — I can implement a higher-level workaround that prevents tools from being injected when tool_choice is set to none, without touching the template itself.

@abdullah-cod9 abdullah-cod9 marked this pull request as draft April 7, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: tool_choice Not Respected in Qwen3.5-VL: Tools Triggered on Every Request

2 participants