fix: protect fenced code blocks and expand MDX includes for agent docs#16
Open
scottamain wants to merge 4 commits into
Open
fix: protect fenced code blocks and expand MDX includes for agent docs#16scottamain wants to merge 4 commits into
scottamain wants to merge 4 commits into
Conversation
The step-9 cleanup regex (/<[A-Z][a-zA-Z]*.../) removes MDX components from raw .md output. Inside bash heredocs, `<<EOF` is misread as a JSX tag named EOF because the pattern only requires an uppercase letter after `<`. On docs.modular.com this corrupted max/inference/image-generation.md: the stripper deleted from `cat <<EOF` through `<InstallOpenAI />`, leaving an unclosed ```bash fence and failing the markdown-code-fence-validity AFD check. Fix: tokenize fenced code blocks before step 9 and restore them afterward.
Self-closing tags like <InstallModular folder="..." /> were stripped entirely from .md output, while the HTML page renders the full install instructions. The afdocs markdown-content-parity check compares Turndown(HTML) against the .md endpoint, so agents saw missing setup steps. Expand _includes/install-modular.mdx and install-openai.mdx inline at build time, converting nested CodeBlock, ConditionalVersionDocs, Admonition, and Tabs markup. Also convert <figure> blocks with require()-based images to markdown with captions. Separate from the fenced-code-block protection fix — that addresses a corruption bug; this improves parity between HTML and .md content. Co-authored-by: Cursor <cursoragent@cursor.com>
…d tests Three improvements to the code-fence protection PR: 1. Restore InstallModular to the unwrapMdxComponents list so that non-self-closing <InstallModular>...</InstallModular> usage (if any site uses it as a wrapper) preserves inner content rather than having it silently deleted by the JSX stripper. The converter runs first and handles self-closing tags; unwrap acts as a safety net for the rest. 2. Cache include template reads in pluginContext. Previously convertInstallModularToMarkdown and convertInstallOpenAIToMarkdown called fs.existsSync + fs.readFileSync on every file processed. Now the templates are read once in postBuild and passed through pluginContext.includeTemplates. 3. Add test coverage (32 tests) for the new functions: fenced block protection round-tripping, the JSX stripper heredoc regression, CodeBlock/Admonition/ConditionalVersionDocs/Figure converters, InstallModular/InstallOpenAI expansion with and without pluginContext, and cleanMarkdownForDisplay compatibility without pluginContext. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR improves parity between rendered HTML docs and the emitted .md output by expanding certain MDX include/components into Markdown, and fixes a JSX-stripping regression that could corrupt fenced code blocks (notably bash heredocs).
Changes:
- Adds a protect/restore mechanism for fenced code blocks to prevent the JSX-stripper regex from deleting heredoc content inside fences.
- Expands specific MDX components/includes (e.g., Install* includes, CodeBlock, ConditionalVersionDocs, Admonition, figure-with-require images) into Markdown during
cleanMarkdownForDisplay. - Adds a standalone Node test script covering code-fence protection and the new converters.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| index.js | Adds fence protection, MDX include/component-to-Markdown conversions, and threads pluginContext through build processing. |
| tests/test-code-fence-protection.js | Adds regression/behavior tests for fence protection and new converters (run via node). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review on PR #16: fence protect/restore no longer injects extra newlines or trims block content, and Admonition conversion blockquotes every line for valid markdown semantics. Co-authored-by: Cursor <cursoragent@cursor.com>
| function protectFencedCodeBlocks(content) { | ||
| const blocks = []; | ||
| const protectedContent = content.replace( | ||
| /(^|\n)(```[\s\S]*?\n```)/g, |
| const includeTemplate = pluginContext.includeTemplates?.['install-modular']; | ||
| if (!includeTemplate) return content; | ||
|
|
||
| return content.replace(/<InstallModular\s+([^>]*?)\/>/gi, (match) => { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things in here:
Self-closing tags like
<InstallModular folder="..." />were stripped entirelyfrom .md output, while the HTML page renders the full install instructions.
The afdocs markdown-content-parity check compares Turndown(HTML) against the
.md endpoint, so agents saw missing setup steps.
This expands them, also converts nested CodeBlock, ConditionalVersionDocs, Admonition, and
blocks with require()-based images toTabs markup. Also convert
markdown with captions.
This improves parity between HTML and .md content.
Cleanup regex
(/<[A-Z][a-zA-Z]*.../)removes MDX components fromraw .md output. Inside bash code blocks,
<<EOFwas misread as a JSX tag namedEOF because the pattern only requires an uppercase letter after
<.On docs.modular.com this corrupted
max/inference/image-generation.md: thestripper deleted from
cat <<EOFthrough<InstallOpenAI />, leaving anunclosed ```bash fence and failing the markdown-code-fence-validity AFD check.