Skip to content

fix(deps): lift msgspec cap to 0.21.1 on py3.10+ for py3.13 concurrent-decode segfault (CNCT-85)#977

Open
fyzanshaik-atlan wants to merge 2 commits into
mainfrom
cnct-85-lift-msgspec-cap
Open

fix(deps): lift msgspec cap to 0.21.1 on py3.10+ for py3.13 concurrent-decode segfault (CNCT-85)#977
fyzanshaik-atlan wants to merge 2 commits into
mainfrom
cnct-85-lift-msgspec-cap

Conversation

@fyzanshaik-atlan

Copy link
Copy Markdown
Contributor

TL;DR

pyatlan caps msgspec<0.21. msgspec 0.20.0 has a native segfault on Python 3.13+ under concurrent JSON decode (msgspec #989 / #960), fixed in 0.21.0/0.21.1. SDK >=3.23.0 validates transformed assets by decoding pyatlan assets concurrently on every upload(), so on py3.13 it hits the msgspec bug and hard-crashes the worker. Because pyatlan pins below the fix, apps can only adopt it via a per-app override-dependencies hack (atlan-mssql-app PR #214). This lifts the cap so the fix is available cleanly.

Issue: CNCT-85.

What changed

- "msgspec~=0.20.0",
+ "msgspec~=0.20.0; python_version < '3.10'",
+ "msgspec~=0.21.1; python_version >= '3.10'",

A marker split, not a blanket bump. pyatlan supports and CI-tests Python 3.9, but msgspec 0.21's floor is 3.10 — a straight bump would silently drop 3.9. So 3.9 stays on 0.20.x (it can't hit the py3.13 segfault anyway) and everything >=3.10 gets 0.21.1.

Side benefit: py3.14 previously resolved to no msgspec at all (0.20 had no 3.14 wheels), which left pyatlan_v9 unusable there. 0.21.1 covers 3.10-3.14, so this fixes 3.14 too.

Commits

  1. chore(deps): sync requirements.txt with uv.lock — the committed uv.lock was already current (cryptography 49.0.0, <3.15 markers) but requirements.txt was stale (46.0.5, <3.14). Regenerating it for the msgspec change unavoidably corrects that pre-existing drift, so it's isolated in its own commit to keep commit 2 surgical.
  2. fix(deps): lift msgspec cap to 0.21.1 on py3.10+ ... — the pyproject split above, plus uv.lock (adds the 0.21.1 entry) and the 2 msgspec lines in requirements.txt.

Verification

  • Full v9 unit suite: 7190 passed, 5 skipped, 0 failed on 0.21.1.
  • Exercised the exact crash frames — Column.from_json -> serde.decode(ColumnNested) through the shared Serde singleton, 2000 concurrent decodes — all correct.
  • uv lock --check consistent.

Caveat: this proves API compatibility. The segfault itself only reproduces on native Linux x86 under concurrent load (not macOS arm64), so the fix's efficacy rests on the msgspec #960 change plus the mssql-app CI repro.

Follow-ups (not in this PR)

  • application-sdk: interim, flip VALIDATE_ASSETS_ON_UPLOAD default to false until the floor is safe; after this releases, bump the SDK's pyatlan floor and flip it back on.
  • atlan-mssql-app: drop the override-dependencies once this is released.

@linear

linear Bot commented Jul 17, 2026

Copy link
Copy Markdown

CNCT-85

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant