fix(deps): lift msgspec cap to 0.21.1 on py3.10+ for py3.13 concurrent-decode segfault (CNCT-85)#977
Open
fyzanshaik-atlan wants to merge 2 commits into
Open
fix(deps): lift msgspec cap to 0.21.1 on py3.10+ for py3.13 concurrent-decode segfault (CNCT-85)#977fyzanshaik-atlan wants to merge 2 commits into
fyzanshaik-atlan wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
pyatlan caps
msgspec<0.21. msgspec 0.20.0 has a native segfault on Python 3.13+ under concurrent JSON decode (msgspec #989 / #960), fixed in 0.21.0/0.21.1. SDK >=3.23.0 validates transformed assets by decoding pyatlan assets concurrently on everyupload(), so on py3.13 it hits the msgspec bug and hard-crashes the worker. Because pyatlan pins below the fix, apps can only adopt it via a per-appoverride-dependencieshack (atlan-mssql-app PR #214). This lifts the cap so the fix is available cleanly.Issue: CNCT-85.
What changed
A marker split, not a blanket bump. pyatlan supports and CI-tests Python 3.9, but msgspec 0.21's floor is 3.10 — a straight bump would silently drop 3.9. So 3.9 stays on 0.20.x (it can't hit the py3.13 segfault anyway) and everything >=3.10 gets 0.21.1.
Side benefit: py3.14 previously resolved to no msgspec at all (0.20 had no 3.14 wheels), which left pyatlan_v9 unusable there. 0.21.1 covers 3.10-3.14, so this fixes 3.14 too.
Commits
chore(deps): sync requirements.txt with uv.lock— the committeduv.lockwas already current (cryptography 49.0.0,<3.15markers) butrequirements.txtwas stale (46.0.5,<3.14). Regenerating it for the msgspec change unavoidably corrects that pre-existing drift, so it's isolated in its own commit to keep commit 2 surgical.fix(deps): lift msgspec cap to 0.21.1 on py3.10+ ...— the pyproject split above, plusuv.lock(adds the 0.21.1 entry) and the 2 msgspec lines inrequirements.txt.Verification
Column.from_json->serde.decode(ColumnNested)through the sharedSerdesingleton, 2000 concurrent decodes — all correct.uv lock --checkconsistent.Caveat: this proves API compatibility. The segfault itself only reproduces on native Linux x86 under concurrent load (not macOS arm64), so the fix's efficacy rests on the msgspec #960 change plus the mssql-app CI repro.
Follow-ups (not in this PR)
VALIDATE_ASSETS_ON_UPLOADdefault tofalseuntil the floor is safe; after this releases, bump the SDK's pyatlan floor and flip it back on.override-dependenciesonce this is released.