Skip to content

feat: add LiteLLM as AI gateway prompt target#2154

Open
RheagalFire wants to merge 1 commit into
microsoft:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway prompt target#2154
RheagalFire wants to merge 1 commit into
microsoft:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Description

Adds LiteLLMChatTarget - a new prompt target that uses the LiteLLM SDK to access 100+ LLM providers (Anthropic, AWS Bedrock, Google Vertex, Cohere, etc.) directly, without requiring a separate proxy server.

from pyrit.prompt_target import LiteLLMChatTarget

# Anthropic (reads ANTHROPIC_API_KEY from env)
target = LiteLLMChatTarget(model_name="anthropic/claude-sonnet-4-6")

# AWS Bedrock (reads AWS credentials from env)
target = LiteLLMChatTarget(model_name="bedrock/anthropic.claude-v2")

# Google Vertex AI
target = LiteLLMChatTarget(model_name="vertex_ai/gemini-pro")

# Self-hosted LiteLLM proxy
target = LiteLLMChatTarget(
    model_name="anthropic/claude-sonnet-4-6",
    api_base="http://localhost:4000",
    api_key="sk-proxy-key",
)

Changes:

  • pyrit/prompt_target/litellm_chat_target.py - New LiteLLMChatTarget extending PromptTarget with:
    • litellm.acompletion() for async chat completions
    • drop_params=True by default (silently drops provider-unsupported kwargs for cross-provider compatibility)
    • Exception translation: litellm.exceptions.* mapped to PyRIT's RateLimitException / PyritException hierarchy (no bare Exception catches)
    • Token usage capture in response metadata
    • Tool call / function call support
    • Lazy import litellm so users without the package are unaffected
  • pyrit/prompt_target/__init__.py - Registered LiteLLMChatTarget
  • pyproject.toml - Added litellm>=1.83.0,<2.0.0 as optional dependency (pip install pyrit[litellm])

Tests and Documentation

22 unit tests in tests/unit/prompt_target/target/test_litellm_chat_target.py:

  • Constructor validation: missing model raises ValueError, LITELLM_MODEL env var fallback, explicit override
  • drop_params=True default + opt-out with drop_params=False
  • Request body construction: API key forwarded when set, omitted when blank (lets LiteLLM read provider env vars); optional params (temperature, top_p, max_tokens, api_base) forwarded correctly
  • Full request-response cycle: text responses parsed, tool calls serialized as JSON, token usage captured in metadata
  • Empty/malformed response handling: no choices raises EmptyResponseException, empty content raises EmptyResponseException
  • Exception translation: litellm.exceptions.RateLimitError -> RateLimitException, AuthenticationError -> PyritException, APIConnectionError/Timeout -> retryable RateLimitException, unknown errors -> PyritException
  • Finish reason validation: stop, length, tool_calls, content_filter accepted; unknown reasons raise PyritException
  • Message building: role preservation across multi-turn conversations

All 22 tests pass:

tests/unit/prompt_target/target/test_litellm_chat_target.py  22 passed in 2.15s

Ruff lint clean (line-length=120, preview=true).

JupyText not applicable - no notebook documentation added for this target yet.

@rlundeen2 rlundeen2 self-assigned this Jul 9, 2026
@rlundeen2

rlundeen2 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

I've been wanting to do this also; @RheagalFire this is a great start. I'll likely push to your branch to flush it out and support all the things (multi-modal, identifiers, underlying model, integration tests, capabilities detection, other gaps). So from your perspective I would consider this "merged" and I'll take it and try to get it in before the next release.

TY for the help and nudge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants