Add RelayShield security tools (MCP registry risk + prompt-injection breach detection)#6550
Open
nzdsf2-gif wants to merge 3 commits into
Open
Add RelayShield security tools (MCP registry risk + prompt-injection breach detection)#6550nzdsf2-gif wants to merge 3 commits into
nzdsf2-gif wants to merge 3 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughRelayShield integrations add typed inputs and authenticated API calls for MCP registry risk and prompt-injection breach checks, with formatted results, public package exports, and setup documentation. ChangesRelayShield security tools
Sequence Diagram(s)sequenceDiagram
participant Agent
participant RelayShieldTool
participant RelayShieldAPI
Agent->>RelayShieldTool: provide MCP identifier or email
RelayShieldTool->>RelayShieldAPI: send authenticated risk request
RelayShieldAPI-->>RelayShieldTool: return wrapped assessment
RelayShieldTool-->>Agent: return formatted verdict or exposure result
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py`:
- Around line 2-13: Remove the ClassVar import and its annotation from the
module-level API_BASE_URL constant. Keep API_BASE_URL typed as a regular
module-level string constant.
- Around line 16-26: The keyless x402 route must work without
RELAYSHIELD_API_KEY. In relayshield_tool.py lines 16-26, update
_relayshield_headers to omit X-RS-API-KEY when no key is configured instead of
raising ValueError; also set env_vars required=False at lines 41-47 and 96-102
so both tool configurations permit keyless use.
- Around line 59-71: The RelayShield tools leave response JSON parsing outside
their request error handling, allowing invalid 2XX responses to crash. In
lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py
lines 59-71, update RelayShieldMCPRiskTool to parse the response and extract
data inside the existing try block; apply the same change to
RelayShieldPromptInjectionBreachTool at lines 105-117, preserving the existing
failure-string handling for parsing errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5365f4f2-4c24-4950-bd27-d9131f8bff99
📒 Files selected for processing (5)
lib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/relayshield_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/relayshield_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.pylib/crewai-tools/src/crewai_tools/tools/relayshield_tool/schemas.py
Comment on lines
+16
to
+26
| def _relayshield_headers() -> dict[str, str]: | ||
| api_key = os.environ.get("RELAYSHIELD_API_KEY", "") | ||
| if not api_key: | ||
| raise ValueError( | ||
| "RELAYSHIELD_API_KEY environment variable is required. " | ||
| "Get a key at https://api.relayshield.net/developers" | ||
| ) | ||
| # Note: relayshield_agentic_api.py (which hosts these two endpoints) | ||
| # expects X-RS-API-KEY specifically, not the X-API-Key convention used | ||
| # by RelayShield's other metered endpoints. | ||
| return {"Content-Type": "application/json", "X-RS-API-KEY": api_key} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
API key requirement breaks the keyless x402 pay-per-call route. The PR objectives and documentation state that the tools support an x402 route requiring no API key. However, the code strictly enforces the presence of RELAYSHIELD_API_KEY, which will block users attempting to use the keyless payment route.
lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L16-L26: Remove theValueErrorwhenapi_keyis empty, and instead omit theX-RS-API-KEYheader if a key isn't provided.lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L41-L47: Changerequired=Truetorequired=Falsein theenv_varsdeclaration.lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L96-L102: Changerequired=Truetorequired=Falsein theenv_varsdeclaration.
🛠️ Proposed fix for `_relayshield_headers`
def _relayshield_headers() -> dict[str, str]:
+ headers = {"Content-Type": "application/json"}
api_key = os.environ.get("RELAYSHIELD_API_KEY", "")
- if not api_key:
- raise ValueError(
- "RELAYSHIELD_API_KEY environment variable is required. "
- "Get a key at https://api.relayshield.net/developers"
- )
- # Note: relayshield_agentic_api.py (which hosts these two endpoints)
- # expects X-RS-API-KEY specifically, not the X-API-Key convention used
- # by RelayShield's other metered endpoints.
- return {"Content-Type": "application/json", "X-RS-API-KEY": api_key}
+ if api_key:
+ headers["X-RS-API-KEY"] = api_key
+ return headers📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _relayshield_headers() -> dict[str, str]: | |
| api_key = os.environ.get("RELAYSHIELD_API_KEY", "") | |
| if not api_key: | |
| raise ValueError( | |
| "RELAYSHIELD_API_KEY environment variable is required. " | |
| "Get a key at https://api.relayshield.net/developers" | |
| ) | |
| # Note: relayshield_agentic_api.py (which hosts these two endpoints) | |
| # expects X-RS-API-KEY specifically, not the X-API-Key convention used | |
| # by RelayShield's other metered endpoints. | |
| return {"Content-Type": "application/json", "X-RS-API-KEY": api_key} | |
| def _relayshield_headers() -> dict[str, str]: | |
| headers = {"Content-Type": "application/json"} | |
| api_key = os.environ.get("RELAYSHIELD_API_KEY", "") | |
| if api_key: | |
| headers["X-RS-API-KEY"] = api_key | |
| return headers |
📍 Affects 1 file
lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L16-L26(this comment)lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L41-L47lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L96-L102
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py`
around lines 16 - 26, The keyless x402 route must work without
RELAYSHIELD_API_KEY. In relayshield_tool.py lines 16-26, update
_relayshield_headers to omit X-RS-API-KEY when no key is configured instead of
raising ValueError; also set env_vars required=False at lines 41-47 and 96-102
so both tool configurations permit keyless use.
…-key crash - Remove ClassVar from module-level API_BASE_URL (only valid in class bodies) - Move resp.json() parsing inside the try block so an invalid-JSON 2XX response is caught as a RequestException instead of crashing - Check for RELAYSHIELD_API_KEY before the request and return a clean error string instead of raising an uncaught ValueError from inside the try block Not implementing full keyless x402 client signing per the "omit the header" suggestion — these tools only call the metered endpoint, which has no x402 signing logic behind it; omitting the header would just produce a 401, not enable PAYG. That's a larger scope change than this PR, not a quick fix.
Author
|
Thanks for the review — addressed in fd68030:
Not implementing the "omit the header, allow keyless" suggestion as proposed: these tools only call the metered endpoint ( Smoke-tested all three cases (successful call, breach check, missing-key error path) against the live API before pushing. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py`:
- Around line 106-111: Add the missing os module import to the module containing
the RelayShield tool before its use in the API-key validation flow, so the
existing os.environ.get call works without changing the validation behavior.
- Around line 47-52: Add the missing os module import at the top of the module
containing the RelayShield tool before the api_key lookup in the tool’s
initialization or execution flow. Preserve the existing RELAYSHIELD_API_KEY
validation and error response.
- Around line 68-69: Validate that the parsed JSON is a dictionary before
calling .get() in both RelayShieldMCPRiskTool
(lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L68-L69)
and RelayShieldPromptInjectionBreachTool
(lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py#L121-L122).
For non-dictionary responses, return the tool’s existing clean error string
instead of allowing AttributeError to escape; preserve normal data extraction
for dictionary responses.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 41aa096f-5bea-4422-a3a8-f05a451b9cac
📒 Files selected for processing (1)
lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py
Comment on lines
+47
to
+52
| api_key = os.environ.get("RELAYSHIELD_API_KEY", "") | ||
| if not api_key: | ||
| return ( | ||
| "Error: RELAYSHIELD_API_KEY environment variable is required. " | ||
| "Get a key at https://api.relayshield.net/developers" | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Missing os module import.
The os module is used here to read RELAYSHIELD_API_KEY, but based on the file context, it has not been imported. This will raise a NameError at runtime.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py`
around lines 47 - 52, Add the missing os module import at the top of the module
containing the RelayShield tool before the api_key lookup in the tool’s
initialization or execution flow. Preserve the existing RELAYSHIELD_API_KEY
validation and error response.
Comment on lines
+106
to
+111
| api_key = os.environ.get("RELAYSHIELD_API_KEY", "") | ||
| if not api_key: | ||
| return ( | ||
| "Error: RELAYSHIELD_API_KEY environment variable is required. " | ||
| "Get a key at https://api.relayshield.net/developers" | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Missing os module import (mirrors the risk tool).
The os module is used here to read RELAYSHIELD_API_KEY, but it is not imported, which will result in a NameError.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/relayshield_tool/relayshield_tool.py`
around lines 106 - 111, Add the missing os module import to the module
containing the RelayShield tool before its use in the API-key validation flow,
so the existing os.environ.get call works without changing the validation
behavior.
resp.json() could theoretically return a valid-but-non-dict JSON value (list, string, etc.) for a 2XX response, which would crash with AttributeError on .get() before ever reaching the RequestException handler. Now checks isinstance(response_json, dict) before extracting data.
Author
|
Addressed in dd3ac7b:
Re-tested both the real API call and a mocked non-dict-response case before pushing — both behave correctly now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two tools for agent-specific security checks: mcp-registry-risk (typosquat/reputation scoring for MCP servers and tool registries) and prompt-injection-breach (credential breach exposure sourced from prompt-injection attacks against agents). Both call RelayShield's live API — no API key required via the x402 PAYG route (pay-per-call in USDC, no signup), or an API key for the metered route. Docs/usage example included in the tool's README following the existing pattern.