Summary
With SKILLSPECTOR_PROVIDER=openai_compatible pointed at an endpoint that ignores both response_format and a forced tool_choice, every semantic analyzer fails structured-response validation and the report falls back to static analysis. iFlytek's Astron Token Plan (https://maas-token-api.cn-huabei-1.xf-yun.com/v2, model spark-x2.5) is one such endpoint.
The bedrock provider already handles this class of model: a registry entry with tool_choice: auto binds the schema as a tool without forcing it, asks for the tool call in the prompt, and retries a prose answer (bind_structured_output → _require_tool_call). openai_compatible has no way to opt into that path. Its ChatOpenAI always uses LangChain's default json_schema method, and SKILLSPECTOR_STRUCTURED_OUTPUT_METHOD=function_calling still sends a forced tool_choice with no prompt instruction, so the endpoint answers in prose and the parser returns None.
Reproduction
export SKILLSPECTOR_PROVIDER=openai_compatible
export SKILLSPECTOR_COMPAT_BASE_URL=https://maas-token-api.cn-huabei-1.xf-yun.com/v2
export SKILLSPECTOR_COMPAT_API_KEY=...
export SKILLSPECTOR_MODEL=spark-x2.5
skillspector scan tests/fixtures/malicious_skill --format json --output report.json
On main (2226747):
WARNING LLM structured response validation failed for File: SKILL.md after 4 attempts
WARNING LLM stage degraded: semantic runtime telemetry was incomplete; report may reflect static analysis only
analysis_completeness reports status: partial with coverage_percent: 0, and llm_structured_response_invalid for semantic_developer_intent, semantic_security_discovery and semantic_quality_policy. Only the 7 static findings are reported.
Probing the endpoint directly with LangChain shows why:
| method |
request |
response |
json_schema (default) |
response_format: json_schema |
prose (Yes 10/10), so ValidationError |
json_mode |
response_format: json_object |
prose, so OutputParserException |
function_calling |
forced tool_choice |
prose, no tool_calls, so parsed None |
tools with tool_choice auto + "call the tool" instruction |
|
tool_calls with valid args (3/3) |
Proposal
Let openai_compatible honour the same registry key the Bedrock provider uses:
tool_choice: auto in the provider registry (bundled, or SKILLSPECTOR_MODEL_REGISTRY) builds ChatOpenAI with disabled_params={"tool_choice": None}, so LangChain's function_calling method binds the tool without forcing it, and selects function_calling as the structured-output method.
bind_structured_output treats a chat model with tool_choice disabled like Bedrock's supports_tool_choice_values=("auto",), so it gets the prompt instruction and the fail-closed retry.
- Bundle a
spark-x2.5 entry (context_length: 262144, tool_choice: auto) in the openai_compatible registry.
Other providers and models are unchanged, and an explicit structured_output: registry value or SKILLSPECTOR_STRUCTURED_OUTPUT_METHOD still wins. I have a patch with unit tests and a before/after live scan, and will open a PR referencing this issue.
Summary
With
SKILLSPECTOR_PROVIDER=openai_compatiblepointed at an endpoint that ignores bothresponse_formatand a forcedtool_choice, every semantic analyzer fails structured-response validation and the report falls back to static analysis. iFlytek's Astron Token Plan (https://maas-token-api.cn-huabei-1.xf-yun.com/v2, modelspark-x2.5) is one such endpoint.The
bedrockprovider already handles this class of model: a registry entry withtool_choice: autobinds the schema as a tool without forcing it, asks for the tool call in the prompt, and retries a prose answer (bind_structured_output→_require_tool_call).openai_compatiblehas no way to opt into that path. ItsChatOpenAIalways uses LangChain's defaultjson_schemamethod, andSKILLSPECTOR_STRUCTURED_OUTPUT_METHOD=function_callingstill sends a forcedtool_choicewith no prompt instruction, so the endpoint answers in prose and the parser returnsNone.Reproduction
On
main(2226747):analysis_completenessreportsstatus: partialwithcoverage_percent: 0, andllm_structured_response_invalidforsemantic_developer_intent,semantic_security_discoveryandsemantic_quality_policy. Only the 7 static findings are reported.Probing the endpoint directly with LangChain shows why:
json_schema(default)response_format: json_schemaYes 10/10), soValidationErrorjson_moderesponse_format: json_objectOutputParserExceptionfunction_callingtool_choicetool_calls, so parsedNonetool_choiceauto + "call the tool" instructiontool_callswith valid args (3/3)Proposal
Let
openai_compatiblehonour the same registry key the Bedrock provider uses:tool_choice: autoin the provider registry (bundled, orSKILLSPECTOR_MODEL_REGISTRY) buildsChatOpenAIwithdisabled_params={"tool_choice": None}, so LangChain'sfunction_callingmethod binds the tool without forcing it, and selectsfunction_callingas the structured-output method.bind_structured_outputtreats a chat model withtool_choicedisabled like Bedrock'ssupports_tool_choice_values=("auto",), so it gets the prompt instruction and the fail-closed retry.spark-x2.5entry (context_length: 262144,tool_choice: auto) in theopenai_compatibleregistry.Other providers and models are unchanged, and an explicit
structured_output:registry value orSKILLSPECTOR_STRUCTURED_OUTPUT_METHODstill wins. I have a patch with unit tests and a before/after live scan, and will open a PR referencing this issue.