feat(tools): add Hlido agent trust tools (trust check + recommend)#6554
feat(tools): add Hlido agent trust tools (trust check + recommend)#6554ankitkapur1992-hlido wants to merge 1 commit into
Conversation
…ecommendTool) Two tools that let a crew vet other AI agents against Hlido's independent, evidence-backed reviews before delegating to them: - HlidoTrustCheckTool: score/tier/PASS-FAIL verdict for a known agent slug - HlidoRecommendTool: find a reviewed agent for a free-text need, ranked by score Both use the public Hlido API (no API key) and add no new dependencies. Includes README, mocked unit tests, and tool.specs.json entries.
📝 WalkthroughWalkthroughAdds ChangesHlido tool runtime
Public exports and specifications
Runtime behavior tests
Sequence Diagram(s)sequenceDiagram
participant CrewAI
participant HlidoTools
participant HlidoAPI
CrewAI->>HlidoTools: Run trust check or recommendation
HlidoTools->>HlidoAPI: Request scorecard or review registry
HlidoAPI-->>HlidoTools: Return review data or error
HlidoTools-->>CrewAI: Return verdict or ranked JSON results
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (5)
lib/crewai-tools/tests/tools/hlido_trust_tool_test.py (2)
114-116: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAvoid
ValueErrorto ensure a clearAssertionErroron test failure.Using
.index()will raise aValueErrorif the substring is not found inresult, which disrupts standard test runner tracebacks. Using.find()ensures the test fails with a cleanerAssertionErrorinstead.💡 Proposed refactor
- assert result.index('"slug": "b"') < result.index('"slug": "a"') - assert '"slug": "d"' not in result - assert '"slug": "c"' not in result + pos_b = result.find('"slug": "b"') + pos_a = result.find('"slug": "a"') + assert pos_b != -1 and pos_a != -1 + assert pos_b < pos_a + assert '"slug": "d"' not in result + assert '"slug": "c"' not in result🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/tests/tools/hlido_trust_tool_test.py` around lines 114 - 116, Update the ordering assertion in the relevant test to use non-throwing substring position checks instead of str.index, so missing slugs produce a clear AssertionError through the assertion framework. Preserve the existing requirement that slug “b” appears before slug “a”, while keeping the exclusions for slugs “c” and “d”.
10-15: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winImprove mock realism for HTTP errors and empty JSON lists.
Currently,
raise_for_status()is mocked to returnNoneeven for error status codes (like 404), which does not reflect the actual behavior of therequestslibrary. This could hide bugs if the underlying tool logic relies on catchingHTTPError. Additionally,json_data or {}will incorrectly evaluate to{}ifjson_datais explicitly provided as an empty list[].Consider updating the mock to accurately raise
HTTPErroron error codes and correctly handle falsyjson_datainputs.♻️ Proposed refactor
def _mock_response(status_code=200, json_data=None): + import requests resp = MagicMock() resp.status_code = status_code - resp.json.return_value = json_data or {} - resp.raise_for_status.return_value = None + resp.json.return_value = json_data if json_data is not None else {} + if status_code >= 400: + resp.raise_for_status.side_effect = requests.exceptions.HTTPError(response=resp) + else: + resp.raise_for_status.return_value = None return resp🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/tests/tools/hlido_trust_tool_test.py` around lines 10 - 15, Update the _mock_response helper to make raise_for_status raise requests.exceptions.HTTPError for HTTP error status codes while preserving successful responses, and replace the json_data or {} fallback with a None-specific check so explicitly supplied empty lists remain unchanged.lib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/hlido_trust_tool.py (2)
186-188: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valuePrevent
"None"from being injected into the searchable text.If the Hlido API returns
nullfor a key (e.g.,"summary": null),item.get("summary", "")will returnNone, as the key exists. Callingstr(None)turns it into the string"None", which lowercases to"none". This means the word "none" becomes part of the searchable text, potentially causing false-positive matches.Using
item.get(k) or ""cleanly falls back to an empty string for both missing keys andnullvalues.♻️ Proposed refactor
haystack = " ".join( - str(item.get(k, "")) for k in ("name", "slug", "category", "summary") + str(item.get(k) or "") for k in ("name", "slug", "category", "summary") ).lower()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/hlido_trust_tool.py` around lines 186 - 188, Update the searchable text construction in the Hlido tool’s haystack expression to use an empty-string fallback for both missing and null values before converting items to strings. Specifically adjust the item field lookup in the join over name, slug, category, and summary so null API fields cannot produce the literal “None”.
84-84: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueURL-encode the slug to prevent invalid URL exceptions.
Although the slug is stripped and lowercased, a user might still pass an unexpected string containing spaces or special characters (e.g.,
"aider agent"). Usingurllib.parse.quoteensures the URL remains valid and gracefully falls back to a 404 (handled by your logic below) rather than raising an unhandledrequests.exceptions.InvalidURLexception.💡 Proposed refactor
Ensure you add
import urllib.parseat the top of the file, then apply:- url = f"{self.base_url}/data/scorecards/{slug}.json" + encoded_slug = urllib.parse.quote(slug) + url = f"{self.base_url}/data/scorecards/{encoded_slug}.json"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/hlido_trust_tool.py` at line 84, Update the URL construction in the HlidоTrust tool method around the scorecard slug to URL-encode the normalized slug with urllib.parse.quote before interpolating it into the endpoint. Add the required urllib.parse import and preserve the existing 404 fallback handling for encoded invalid or unexpected slugs.lib/crewai-tools/src/crewai_tools/tools/__init__.py (1)
87-90: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueImport from the package rather than the internal module.
You have correctly set up an
__init__.pyfile inside thehlido_trust_tooldirectory to exportHlidoRecommendToolandHlidoTrustCheckTool. It is more idiomatic to import from the package boundary instead of directly from the internal module file.
lib/crewai-tools/src/crewai_tools/tools/__init__.py#L87-L90: Change the import path fromcrewai_tools.tools.hlido_trust_tool.hlido_trust_tooltocrewai_tools.tools.hlido_trust_tool.lib/crewai-tools/src/crewai_tools/__init__.py#L98-L101: Apply the exact same change to import fromcrewai_tools.tools.hlido_trust_tool.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/__init__.py` around lines 87 - 90, Update the HlidoRecommendTool and HlidoTrustCheckTool imports in lib/crewai-tools/src/crewai_tools/tools/__init__.py lines 87-90 and lib/crewai-tools/src/crewai_tools/__init__.py lines 98-101 to import from the hlido_trust_tool package boundary rather than its internal hlido_trust_tool module, preserving the existing exported symbols.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@lib/crewai-tools/src/crewai_tools/tools/__init__.py`:
- Around line 87-90: Update the HlidoRecommendTool and HlidoTrustCheckTool
imports in lib/crewai-tools/src/crewai_tools/tools/__init__.py lines 87-90 and
lib/crewai-tools/src/crewai_tools/__init__.py lines 98-101 to import from the
hlido_trust_tool package boundary rather than its internal hlido_trust_tool
module, preserving the existing exported symbols.
In
`@lib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/hlido_trust_tool.py`:
- Around line 186-188: Update the searchable text construction in the Hlido
tool’s haystack expression to use an empty-string fallback for both missing and
null values before converting items to strings. Specifically adjust the item
field lookup in the join over name, slug, category, and summary so null API
fields cannot produce the literal “None”.
- Line 84: Update the URL construction in the HlidоTrust tool method around the
scorecard slug to URL-encode the normalized slug with urllib.parse.quote before
interpolating it into the endpoint. Add the required urllib.parse import and
preserve the existing 404 fallback handling for encoded invalid or unexpected
slugs.
In `@lib/crewai-tools/tests/tools/hlido_trust_tool_test.py`:
- Around line 114-116: Update the ordering assertion in the relevant test to use
non-throwing substring position checks instead of str.index, so missing slugs
produce a clear AssertionError through the assertion framework. Preserve the
existing requirement that slug “b” appears before slug “a”, while keeping the
exclusions for slugs “c” and “d”.
- Around line 10-15: Update the _mock_response helper to make raise_for_status
raise requests.exceptions.HTTPError for HTTP error status codes while preserving
successful responses, and replace the json_data or {} fallback with a
None-specific check so explicitly supplied empty lists remain unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 11906207-e113-47e2-b90d-a8a7bd28b7ee
📒 Files selected for processing (7)
lib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/hlido_trust_tool.pylib/crewai-tools/tests/tools/hlido_trust_tool_test.pylib/crewai-tools/tool.specs.json
1 participant
What
Adds two CrewAI tools under
lib/crewai-tools/src/crewai_tools/tools/hlido_trust_tool/that let a crew vet other AI agents against independent, evidence-backed reviews before delegating work to them:HlidoTrustCheckTool— given an agent slug (e.g.aider), returns its Hlido score (0-100), tier (VITAL/STEADY/FADING/FLATLINE), aPASS/FAILgate against amin_score, and the reviewed strengths / red flags.HlidoRecommendTool— given a free-text need (e.g. "AI coding agent"), returns matching reviewed agents ranked by trust score.Why
Agents increasingly delegate to other agents/tools with no independent signal of reliability. These tools give a routing/orchestration agent a machine-readable trust check at decision time — a natural fit for CrewAI's delegation model.
Notes
hlido.eu).requests(already a CrewAI dependency).args_schema, graceful error strings (no exceptions leaking to the agent), README, and mocked unit tests (no network).crewai_tools/tools/__init__.pyandcrewai_tools/__init__.py;tool.specs.jsonregenerated (only the two new entries added).uv run pytest tests/tools/hlido_trust_tool_test.py— 8 passed;black/isortclean.Disclosure
I'm the founder of Hlido. This integration is opt-in, uses only the public Hlido API, and adds no new dependencies. Happy to adjust naming/placement to match maintainer preferences.