feat: Add native NVIDIA NIM provider

[88plug]

Overview

This PR introduces native support for NVIDIA NIM (NVIDIA Inference Microservices), providing direct access to 180+ high-performance AI models including Qwen, LLaMA, DeepSeek R1, and Mistral.

Why Native Provider: LiteLLM does not fully support NVIDIA's specialized model types—reasoning models (DeepSeek R1 with chain-of-thought), vision models (Llama 3.2 Vision, VILA), and structured outputs require NVIDIA-specific handling. When NVIDIA_API_KEY is set, the native provider bypasses LiteLLM entirely, connecting directly to NVIDIA's API for guaranteed compatibility and lower latency. Automatically discovers new models via live catalog API, ensuring new NVIDIA releases work immediately without code updates.

Community Impact: This native integration bridges the NVIDIA AI ecosystem with CrewAI's multi-agent framework, enabling NVIDIA developers to now build production-ready agent systems using NVIDIA's models. With free API access at https://build.nvidia.com/ (no credit card required), both communities gain immediate access to state-of-the-art models for education, research, prototyping, and production—fostering cross-pollination between NVIDIA's model ecosystem and CrewAI's agent orchestration platform.

Key Features

• Auto-Detection: Models with "/" in the name (e.g., qwen/qwen3-next-80b-a3b-instruct) automatically route to NVIDIA provider
• 180+ Models: Chat, code, reasoning, vision, and safety models
• Streaming Support: Real-time response streaming with async/await
• Vision Models: Llama 3.2 Vision (11B/90B), Phi-4 Vision
• Reasoning Models: DeepSeek R1, QwQ-32B with chain-of-thought
• Tool Calling: OpenAI-compatible function calling
• Built-in Security: Input validation, API key sanitization, resource management

Implementation Details

Security Enhancements

• API key sanitization in all error messages
• Input validation with regex pattern to prevent injection attacks
• 1-hour cache TTL to prevent cache poisoning
• Thread-safe operations with proper locking
• Resource cleanup via close(), del(), and context manager

Architecture

• Native provider implementation (not LiteLLM wrapper)
• Automatic routing based on model name pattern
• Backward compatible - zero breaking changes
• Opt-in via NVIDIA_API_KEY environment variable

Files Changed

• lib/crewai/src/crewai/llm.py - NVIDIA model catalog with security fixes
• lib/crewai/src/crewai/llms/providers/nvidia/completion.py - Main provider (1,499 lines)
• docs/en/learn/llm-connections.mdx - User documentation (+58 lines)
• Embedding support and constants

Total: 11 files, +1,993 insertions, -31 deletions

Testing

Comprehensive testing performed:

• ✅ 10/10 real execution tests with actual NVIDIA API calls
• ✅ Single agent tasks
• ✅ Multi-agent sequential crews
• ✅ Tool-using agents
• ✅ Reasoning models (DeepSeek R1)
• ✅ Vision models (Llama 3.2)
• ✅ Code generation (Qwen Coder)
• ✅ 4-agent chains with context passing

Usage Example

import os
from crewai import Agent, LLM

# Set your NVIDIA API key (get free key at https://build.nvidia.com/)
os.environ["NVIDIA_API_KEY"] = "nvapi-your-key-here"

# Automatic NVIDIA routing with "/" in model name
llm = LLM(model="qwen/qwen3-next-80b-a3b-instruct", temperature=0.7)

agent = Agent(
    role="Research Analyst",
    goal="Analyze data and provide insights",
    backstory="Expert in data analysis",
    llm=llm
)

Backward Compatibility

• ✅ Zero breaking changes to existing APIs
• ✅ Other providers (OpenAI, Anthropic) unaffected
• ✅ Graceful fallback to LiteLLM for unknown models
• ✅ All existing tests continue to pass

Additional Notes

• Get free API key at https://build.nvidia.com/
• Set NVIDIA_API_KEY environment variable
• See documentation for complete model catalog and examples

---

Note

Adds first-class NVIDIA NIM integration and routes eligible models natively, bypassing LiteLLM when available.

• Native LLM provider: New NvidiaCompletion with OpenAI-compatible calls, streaming (sync/async), tool/function calling, structured outputs, usage tracking, and vision/reasoning model handling
• Auto-routing: LLM.__new__ checks NVIDIA model catalog (cached, thread-safe) and routes "provider/model" names; updates SUPPORTED_NATIVE_PROVIDERS and provider pattern logic (Gemini tightened)
• Security/robustness: API key sanitization, model name validation, HTTP timeouts, resource cleanup, and 1h cache TTL
• Embeddings: New NVIDIA embeddings provider (NvidiaEmbeddingFunction, NvidiaProvider) wired into factory and allowed providers
• Constants/Docs: Adds initial NVIDIA model constants and expands llm-connections.mdx with a "Native NVIDIA Provider" quick start and feature overview
• Tests: Extensive NVIDIA routing/initialization, tool use, params, context window, usage tracking, and crew execution tests

^{Written by Cursor Bugbot for commit d337190. This will update automatically on new commits. Configure here.}

[cursor]

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[lorenzejay]

can we add tests similar to - https://github.com/crewAIInc/crewAI/blob/main/lib/crewai/tests/llms/google/test_google.py

[cursor]

Missing after_llm_call hooks in astream method

Medium Severity

The astream async generator method emits the completion event but never calls _invoke_after_llm_call_hooks to process the response through registered hooks. All other completion methods (_handle_completion, _handle_streaming_completion, _ahandle_completion, _ahandle_streaming_completion) invoke this hook to allow response modification. Users relying on after_llm_call hooks for logging, filtering, or transforming responses will find their hooks silently skipped when using astream.

 

[cursor]

Streaming structured output silently returns empty on failure

High Severity

When structured output parsing fails in streaming methods (_handle_streaming_completion and _ahandle_streaming_completion), they log an error and return an empty string "". In contrast, the non-streaming methods (_handle_completion at line 631 and _ahandle_completion at line 930) raise a ValueError when parsing fails. This inconsistency causes silent failures in streaming mode - callers receive an empty string instead of an exception, leading to incorrect downstream behavior where the application continues with invalid data rather than handling the error properly.

Additional Locations (1)

• lib/crewai/src/crewai/llms/providers/nvidia/completion.py#L1076-L1078

 

[lorenzejay]

missing import here too

from typing import TYPE_CHECKING, Any, Self

[cursor]

Test assertions inconsistent with model name routing logic

Medium Severity

The test assertions for model names are inconsistent with the actual routing logic. test_nvidia_completion_is_used_when_nvidia_provider and test_nvidia_completion_initialization_parameters expect llm.model to be "llama-3.1-70b-instruct" (with the nvidia/ prefix stripped), but the routing logic in llm.py at line 453 sets model_string = model preserving the full model name "nvidia/llama-3.1-70b-instruct" for all models found in the NVIDIA catalog. This contradicts test_nvidia_completion_is_used_when_model_has_slash which correctly expects the full model name "meta/llama-3.1-70b-instruct" to be preserved. The assertions on lines 28 and 176 need to expect the full model name including the prefix.

🔬 Verification Test

Why verification test was not possible: The tests require mocking the NVIDIA API model catalog lookup (_get_nvidia_models), which the test file doesn't do. Without this mock, the tests make real HTTP calls that fail with the fake API key, causing the model to fall back to LiteLLM instead of NvidiaCompletion. I verified the code logic by tracing through the routing in llm.py lines 449-453, which clearly shows model_string = model (full model name preserved) when a model is found in the NVIDIA catalog.

Additional Locations (1)

• lib/crewai/tests/llms/nvidia/test_nvidia.py#L175-L176

 

[cursor]

astream with response_model never yields final structured result

High Severity

The astream async generator method computes structured_json when response_model is provided but never yields it to the caller. After emitting the completion event, the method executes a bare return statement that ends the generator without yielding the final structured result. Callers iterating over astream with a response_model would only receive partial delta content chunks during streaming but never receive the final parsed structured output. This is inconsistent with _handle_streaming_completion and _ahandle_streaming_completion which correctly return structured_json. A yield structured_json is missing before the return statement.

🔬 Verification Test

Why verification test was not possible: This bug requires an actual NVIDIA API call with streaming enabled and a response_model parameter to observe that the final structured JSON is never yielded. The test infrastructure doesn't mock the streaming API responses properly, making it impossible to verify without live API access.

 

[88plug]

can we add tests similar to - https://github.com/crewAIInc/crewAI/blob/main/lib/crewai/tests/llms/google/test_google.py

Added...doing the bot fixes now.