Fix crash when scorer LLM response is blocked by content filter#2072
Open
hannahwestra25 wants to merge 13 commits into
Open
Fix crash when scorer LLM response is blocked by content filter#2072hannahwestra25 wants to merge 13 commits into
hannahwestra25 wants to merge 13 commits into
Conversation
When the scorer's own LLM call is blocked by Azure content filtering, the response contains only an error piece with no text piece. The next() call in _score_value_with_llm_async raises StopIteration, which becomes a RuntimeError inside the async coroutine. Add an explicit check for all-blocked response pieces before the next() call and raise a descriptive BadRequestException instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz
approved these changes
Jun 25, 2026
…orer-content-filter-crash
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
reviewed
Jul 1, 2026
| # Require every piece to be blocked (all) rather than any: a partially blocked | ||
| # response may still carry a usable text piece, so we only bail out when there | ||
| # is no salvageable content to parse. | ||
| if all(piece.is_blocked() for piece in response[0].message_pieces): |
Contributor
There was a problem hiding this comment.
When a piece is blocked, every piece is marked blocked. The piece's converted_value has the error JSON. There is partial content in the metadata, but no non-blocked text piece.
So this should be updated because it will never get partial content.
Contributor
There was a problem hiding this comment.
Good catch, you're right. A content-filter block always comes back as a single fully-blocked error piece (partial content lives in prompt_metadata, not a separate text piece), so the old comment's salvageable text piece reasoning was wrong. Fixed the comment in 0aa44b9 to just say a block yields a single error piece with no parseable text. The all() check is functionally equivalent to checking the first piece here; kept it as a harmless defensive check rather than changing behavior.
A content-filter block always yields a single fully-blocked error piece, so the previous comment's claim about a salvageable text piece was inaccurate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…scoring The scoring transport was refactored into _run_llm_scoring_async (pyrit/score/llm_scoring.py) with parsing delegated to ResponseHandler. Move the all-pieces-blocked BadRequestException guard from the old inline _score_value_with_llm_async body into the new transport helper, right before the text-piece extraction that previously crashed on fully-blocked responses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
I'm going to work on this and take it over since Hannah is out |
When the scorer's own LLM response is content-filtered, the transport now raises a dedicated ScorerLLMResponseBlockedException instead of a generic BadRequestException. The raise-vs-fallback policy lives in the Scorer (per framework.md): score_async catches the exception and, when raise_if_scorer_blocks is False, returns the scorer's type default (False / 0.0) via _build_fallback_score instead of raising. Defaults to raising. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…m policy Use 'scorer's LLM response' consistently in llm_scoring.py and drop the Scorer.raise_if_scorer_blocks reference from the transport docstring/comment so the transport doesn't name the Scorer's policy attribute. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The crash this PR fixes is a StopIteration when no text piece exists in the scorer's response. The previous guard keyed on 'all pieces blocked', which left two gaps: an empty piece list spuriously raised (all() over empty is True), and a non-blocked response with no text piece fell through and crashed again. Resolve the text piece safely and branch on why it's missing: a content-filter block raises ScorerLLMResponseBlockedException (routed through the Scorer's raise_if_scorer_blocks policy), while any other no-text response raises EmptyResponseException so a genuinely empty/malformed response surfaces as a real error instead of being silently converted to a default score. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…orer-content-filter-crash
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the scorer's own LLM call is blocked by Azure content filtering (e.g., because it's grading harmful content), the response contains only an error piece with no text piece. The next() generator in _score_value_with_llm_async raises StopIteration, which becomes a RuntimeError inside the async coroutine.
This adds an explicit check before the next() call and raises a descriptive BadRequestException suggesting the user configure a scorer endpoint without content filtering.