Skip to content

openai_compatible: no way to use auto tool_choice for endpoints that ignore response_format and forced tool_choice (e.g. iFlytek spark-x2.5) #707

Description

@FenjuFu

Summary

With SKILLSPECTOR_PROVIDER=openai_compatible pointed at an endpoint that ignores both response_format and a forced tool_choice, every semantic analyzer fails structured-response validation and the report falls back to static analysis. iFlytek's Astron Token Plan (https://maas-token-api.cn-huabei-1.xf-yun.com/v2, model spark-x2.5) is one such endpoint.

The bedrock provider already handles this class of model: a registry entry with tool_choice: auto binds the schema as a tool without forcing it, asks for the tool call in the prompt, and retries a prose answer (bind_structured_output → _require_tool_call). openai_compatible has no way to opt into that path. Its ChatOpenAI always uses LangChain's default json_schema method, and SKILLSPECTOR_STRUCTURED_OUTPUT_METHOD=function_calling still sends a forced tool_choice with no prompt instruction, so the endpoint answers in prose and the parser returns None.

Reproduction

export SKILLSPECTOR_PROVIDER=openai_compatible
export SKILLSPECTOR_COMPAT_BASE_URL=https://maas-token-api.cn-huabei-1.xf-yun.com/v2
export SKILLSPECTOR_COMPAT_API_KEY=...
export SKILLSPECTOR_MODEL=spark-x2.5
skillspector scan tests/fixtures/malicious_skill --format json --output report.json

On main (2226747):

WARNING LLM structured response validation failed for File: SKILL.md after 4 attempts
WARNING LLM stage degraded: semantic runtime telemetry was incomplete; report may reflect static analysis only

analysis_completeness reports status: partial with coverage_percent: 0, and llm_structured_response_invalid for semantic_developer_intent, semantic_security_discovery and semantic_quality_policy. Only the 7 static findings are reported.

Probing the endpoint directly with LangChain shows why:

method request response
json_schema (default) response_format: json_schema prose (Yes 10/10), so ValidationError
json_mode response_format: json_object prose, so OutputParserException
function_calling forced tool_choice prose, no tool_calls, so parsed None
tools with tool_choice auto + "call the tool" instruction tool_calls with valid args (3/3)

Proposal

Let openai_compatible honour the same registry key the Bedrock provider uses:

  • tool_choice: auto in the provider registry (bundled, or SKILLSPECTOR_MODEL_REGISTRY) builds ChatOpenAI with disabled_params={"tool_choice": None}, so LangChain's function_calling method binds the tool without forcing it, and selects function_calling as the structured-output method.
  • bind_structured_output treats a chat model with tool_choice disabled like Bedrock's supports_tool_choice_values=("auto",), so it gets the prompt instruction and the fail-closed retry.
  • Bundle a spark-x2.5 entry (context_length: 262144, tool_choice: auto) in the openai_compatible registry.

Other providers and models are unchanged, and an explicit structured_output: registry value or SKILLSPECTOR_STRUCTURED_OUTPUT_METHOD still wins. I have a patch with unit tests and a before/after live scan, and will open a PR referencing this issue.

Activity

  1. rng1995 commented on Oct 4, 2026

    @rng1995
    Collaborator

    Resolution verified: PR #708 has merged. The OpenAI-compatible provider now honors configured tool_choice: auto for the function-calling structured-output path instead of forcing a tool call. The corresponding provider/LLM utility regressions pass on current main. This issue was automatically closed by the merge; broader fenced/prose JSON tolerance in #69 remains separate.

  2. added a commit that references this issue on Oct 4, 2026
    cc49361
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions