HyperTools 1.0: architecture refactor + bug hunt

[jeremymanning]

HyperTools 1.0 — architecture refactor + bug hunt

This PR merges the completed HyperTools 1.0 class-based refactor into dev-1.0, together with a broad bug hunt (open-issue triage + two animation-rendering fixes you reported + a legend-clipping fix) and the CI fixes needed to get all platforms green.

Note: originally developed under the working name "HyperTools 2.0"; renumbered to 1.0 (package version 1.0.0.dev0, branches dev-1.0/dev-1.0-refactor) per project decision. Older commit messages/notes may still say 2.0.

⚠️ Please do not merge. Opened for your review; I'll proceed however you decide. Base: dev-1.0 ← Head: dev-1.0-refactor.

Supersedes #271, which GitHub auto-closed when its head branch was renamed dev-2.0-refactor → dev-1.0-refactor; all commits, evidence, and CI history carry over unchanged.

---

1. Architecture refactor (Plans 1–8)

Reorganized the working dev-1.0 code into the class-based structure, on modern deps, with DataGeometry removed from the public API:

• Deps modernized: numpy ≥2, pandas ≥3, scikit-learn ≥1.4, matplotlib ≥3.8, plotly 6.x, via datawrangler 0.5 (funnel/stack/unstack/format-detection/model-dispatch). Supports pandas 3.0+.
• Class-based modules: core (eval-free apply_model), external (vendored PPCA + brainiak SRM/DetSRM/RSRM), manip (Normalize/ZScore/Smooth/Resample), reduce/align/cluster (generic sklearn dispatch by name), io, plot (matplotlib + plotly backends). Old tools/ names remain as shims.
• DataGeometry removed from the public API (kept only as a hidden unpickle shim so legacy hosted .geo datasets still load). plot() returns a Figure / (fig, ani); plot(..., return_model=True) returns {'fig','xform_data','models','animation'}; load() returns raw data.
• Docs/gallery/tutorials migrated to the 1.0 API and rebuilt.

---

2. Reported animation fixes

Animated bounding box "crowded / cut off on the right." Animated 3D plots zoomed the cube in too far. Fix: set_box_aspect zoom 1.25→1.125 + full-canvas axes. Min cube-to-edge margin over the full save_movie rotation: right 80→96px, bottom 51→72px.

before (crowded)	after (margin)

Duplicate animation-legend entries (GH #207). Faint "trail" artists carried the dataset label, so each label appeared twice. Fix: trails tagged _nolegend_; legend is now the static union of in-focus datasets. Verified ['first','second'].

Clipped static gallery legends. Wide legends (long labels / many entries) clipped off the right edge. Root cause: the fit routine measured the legend under seaborn's (narrower) font, but the figure is saved downstream under the default (wider) font. Fix: _fit_right_legend now measures the rasterized pixels under default rcParams and widens the figure (keeping the plot's size) until the legend has a margin. Regenerated plot_legend/plot_PPCA/plot_missing_data; pixel-based regression test added.

---

3. Open-issue triage → close-on-merge

The triage below has since been fully executed (2026-07-07): all 45 addressed/obsolete issues are CLOSED with per-issue run-code evidence comments; the 22 remaining feature requests carry research comments + low/medium/high effort labels; 6 issues were migrated from the jeremymanning fork (#273–#278, fork tracker now empty); and 6 residual gaps found during re-verification were fixed on this branch (#94, #141, #199, #206, #209, #244). See the audit summary comment. Originally: triaged all 67 open issues against this branch with real repros (see notes/issues-to-close-on-merge.md): 31 addressed/obsolete + 16 fixed or implemented on this branch (incl. §5's #169/#132 and §6's seven features) = 47 to close on merge; 20 stay open (feature requests / design decisions), each documented.

Bugs fixed from triage (all regression-tested):

#	bug	fix
#259	import hypertools mutated global rcParams['pdf.fonttype']	removed import-time mutation; editable-PDF default set inside manage_backend's snapshot/restore
#223	get_proj crash on 2D labeled plots	guard get_proj; match annotate/update tuple shapes for 2D vs 3D
#146/#190	DBSCAN/MeanShift/OPTICS crash on n_clusters	inject n_clusters only when the model's signature accepts it; register those clusterers
#148	show=False leaked the figure into pyplot	plt.close(fig) for static figures (skipped when the user passed ax, and for animated figures, whose timer must stay alive)
#132	DataFrame columns consumed positionally across datasets	format_data aligns named columns BY NAME to the first dataset's order (warns on reorder); mismatched column sets raise a clear ValueError
#214	wiki-model docstring vs wiki_model key	docstring corrected
—	reduce=<class/instance> → UnboundLocalError	initialize model_params in the custom-estimator branch

---

4. CI fixes (get all platforms green)

The first CI run surfaced two platform issues (unrelated to the features above); both fixed:

• Windows (all Pythons) — collection error. datawrangler 0.5.0 evaluates os.getenv('HOME') at import time to build its data dir; HOME is unset on Windows, so os.path.join(None, …) crashed dw's (and hypertools') import. Fixed hypertools-side by setting HOME=expanduser('~') before importing dw; filed upstream as Import crashes on Windows: config datadir evals os.getenv('HOME') which is None data-wrangler#32, fixed in dw 0.5.1 (released; verified import datawrangler works with HOME unset). The pydata-wrangler pin is bumped to >=0.5.1; the one-line HOME guard stays as belt-and-suspenders for environments still on 0.5.0.
• macOS/Ubuntu Python 3.11+ — 3 tests. matplotlib 3.11 (Python 3.11+ only) resets a figure's canvas after plt.close() (the disabling the figure doesn't work as intended #148 fix), so tests/callbacks that read fig.canvas.renderer/buffer_rgba() failed. Fixed by guarding the renderer in update_position and rendering the affected tests through an explicit Agg canvas. (savefig after close still works, so users are unaffected.)
• Windows — animated-figure draw crash. On Windows the animate backend switch to TkAgg actually succeeds (headless Linux/mac fall back to Agg), so the disabling the figure doesn't work as intended #148 plt.close() destroyed the FuncAnimation's real Tk timer and any later draw of the returned figure crashed ('NoneType' object has no attribute 'start'). Animated figures are now exempt from the show=False close — the disabling the figure doesn't work as intended #148 complaint was about static figures, and animations need their timer alive for playback.
• Ubuntu 3.12 — screenshot-verification step. The screenshot harness discovered figures via plt.get_fignums(), which the disabling the figure doesn't work as intended #148 close empties; it now uses the figure(s) returned by plot() (13/13 cases pass).

---

5. New: hyp.predict + hyp.impute (resolves GH #169)

Two new modules in the established class-based style (base class + one file per model + funnel dispatcher), integrated into hyp.plot/hyp.analyze like cluster/align:

• hyp.predict(data, model=..., t=...) — timeseries forecasting: Kalman, GaussianProcess, AutoRegressor (any sklearn regressor, recursive multi-step), ARIMA, Laplace (skaters ensemble), Chronos (HuggingFace foundation model, real chronos-t5-tiny test). t follows Use Kalman filter to fill in missing data #169's spec (int steps, or a datetime on time-indexed data — including past-date truncation). One forecast per input dataset, same dimensions, continued index.
• hyp.impute(data, model=...) — missing data: PPCA (default; clean interface over the vendored implementation — format_data's fill now routes through it, behavior-preserving), SimpleImputer/KNNImputer/IterativeImputer, and Kalman, which fills rows where every feature is NaN — the exact gap Use Kalman filter to fill in missing data #169 describes (PPCA cannot).
• return_model=True on both → (result, fitted) matching apply_model's convention; the fitted model can be passed back as model= on new data and is applied without re-estimation (verified: fit on A, forecast/impute B).
• hyp.plot(data, predict='Kalman', t=30) overlays one dashed, low-opacity, same-color forecast tail per dataset (2D + 3D, both backends, no legend duplication, frame always contains the forecasts). impute= selects the missing-data model in the plot/analyze pipeline.
• Dependencies: new [predict] extra (pykalman, statsmodels, skaters) and [predict-hf] (chronos-forecasting); GaussianProcess/AutoRegressor/sklearn imputers work on the base install; friendly ImportErrors otherwise (fresh-venv verified). yfinance is not a dependency — the tutorial self-installs it.
• Tutorials (executed, real data):
  • stock_forecasting.ipynb — scrapes 2y of real Yahoo Finance prices for 4 tickers, backtests all models against a 30-day holdout with an honest MAE/MAPE table (spoiler: ARIMA/Kalman ≈ the naive baseline, as efficient-market theory predicts — the tutorial says so).
  • projectile_kalman.ipynb — a real NBA SportVU jump-shot arc (25 Hz optical tracking): Kalman imputation of 5 fully-occluded frames recovers them to RMSE 0.20 ft vs the recorded truth; forecasting the arc's final 20 frames from the first 30 lands within MAE 4.2 ft.
• Forecast direction fix (review follow-up): the GaussianProcess default kernel is now DotProduct + RBF + WhiteKernel — the old stationary RBF reverted forecasts to the training mean beyond the data (drift −0.026/pt vs observed +0.0019/pt: reversed); the linear term extrapolates trends (+0.002..+0.010/pt: continues). Before/after renders + measurements in this comment.
• Gallery: plot_predict (helical forecasts) + plot_impute (PPCA-vs-Kalman panels on the Use Kalman filter to fill in missing data #169 case); API reference sections added.

---

6. Seven long-standing feature requests (GH #95, #100, #108, #109, #127, #142, #177, #191)

All seven implemented in the 1.0 design language, in both rendering backends, each with numeric + screenshot evidence in this comment:

• support multiindex Pandas DataFrames #95 MultiIndex DataFrames → leaf trajectories + per-level group means (thicker/opaquer means per level, colors by top-level index, exactly per the issue's spec).
• colorbar option #100 colorbar=True/dict — continuous hue (same palette as the lines, real value range) and discrete groups (segmented, labeled); coexists with legends without clipping.
• triangular meshes #109 surface=True/dict — smooth, lit convex-hull surfaces (3D, Blinn-Phong + plotly vertex shading) and smooth filled shapes (2D), dict-controlled properties, animatable (per-frame hull recompute); the axes cube auto-expands to contain the hull. New animate_surface_morph gallery demo.
• support kernel density plots in 2d and 3d #108/enhancement: replace matplotlib with ipyvolume #191 density=True/dict — subtle per-group KDE clouds (off by default; 2D alpha-ramp, 3D iso-surfaces/go.Volume with small-cluster auto-boost); enhancement: replace matplotlib with ipyvolume #191's ipyvolume ask is superseded by the plotly backend.
• feature request: support different selections of bullettime, precog, and chemtrails for different data matrices #127 chemtrails/precog/bullettime accept per-dataset lists (mixed trail styles in one animation); spin/serial warn instead of silently ignoring.
• BUG: legend isn't updated when color kwarg is specified #142 original legend/color symptom already fixed by the refactor (verified); re-verification exposed colors=/linestyles=/markers= were silent no-ops without their singular twins — fixed.
• Add support for Google Drive and other easy-to-add filetypes to hyp.load #177 hyp.load completed: Drive large-file interstitial (verified against a real 498MB public file), .xlsx/.xls, Google Sheets→CSV, remote-pickle trust policy.

Every graphical feature was adversarially screenshot-reviewed by fresh agents across a {2D,3D}×{mpl,plotly}×{static,animated} grid; their findings drove 6 further fix commits (plotly WebGL surface artifacts, bounding-box containment, MultiIndex colorbars, legend fitting, 3D density visibility, trail-mode warnings). New gallery examples: plot_surface, plot_density, plot_colorbar, plot_multiindex, animate_trails_mix, animate_surface_morph.

Maintainer-feedback rounds (tight hulls + morph, constant rotation speed + plotly parity + lighting controls, axes-clipping + gif corruption + full-sample morphs, multibyte text support, GH #205): hulls now hug the observations (hull-hugging smoothing: post-Taubin pull-back to the hull; cube-cloud oversize 1.63×→1.13×, ≥99% containment; axes box sized from actual meshes incl. mid-morph union bound, both backends) and morphing is a first-class animation style — animate='morph' (Hungarian point-cloud morphs between datasets, tagged-list form for static backdrops) with per-segment rotations lists ([1, 0.25, 2, ...] = per hold/transition). Both shape-morph gallery demos collapsed to single hyp.plot calls. Morph rotation speed is constant (segment duration ∝ rotation count); plotly marker sizes are empirically calibrated to matplotlib (15.1px → 5.0px for markersize=6) and volumetric shading retuned to matplotlib's subtlety; every surface color/lighting/shading knob is verified effective on both backends (incl. new lightdir; silent no-op keys removed). The "cut off bounding box" report was traced to two real rendering bugs, both fixed: 3-D scene artists were clipped at matplotlib's aspect-shrunk square viewport (now unclipped in every animation path), and animation GIFs saved at reduced dpi were corrupted by a matplotlib writer resize through the interactive-backend window (saves now dpi-safe). Morph animations use every dataset's full sample set (duplicate-to-largest, duplicates hidden at holds; hold frames are a 100% pixel match to static plots). GH #205 fixed: full multibyte (CJK) text support in both backends — automatic covering-font detection (excluding placeholder fonts like LastResort) + a font= kwarg (family/path/FontProperties) applied to labels/legends/colorbars/titles; plotly also gained real labels= annotations (was a silent no-op); CI provisions CJK fonts on all platforms with anti-tofu pixel tests.

---

Testing

• Local suite: 947 passed, 0 failed (+492 tests across all rounds: surface/density/colorbar/multiindex/trails/meshutil/load/color-alias/morph-animation/hull-tightness suites); 6 plotly→kaleido image-export tests deselected locally only — they deadlock Chromium in this sandbox but run fine in CI.
• CI: all 12 jobs green (ubuntu/macos/windows × Python 3.10–3.13) — see this PR's checks for the head commit.
• Docs rebuilt (make html succeeds); gallery regenerated with the 6 new example pages (including two captured animations).

🤖 Generated with Claude Code

[jeremymanning]

Full issue audit executed: both trackers verified, updated, and organized

Every open issue on ContextLab/hypertools (67) and jeremymanning/hypertools (13) was re-verified against this branch by actually running the code (no verdicts from reading source alone — each issue's comment carries copy-pasted commands + output, and rendered-image checks where the question was visual). Outcomes:

ContextLab (67 issues → 28 remain open, all organized)

• 39 closed as addressed — each with an evidence comment linking this PR. Six of these were made true during the audit itself (verification found residual gaps, which were fixed on this branch rather than hand-waved):
  • bug: animation indexing error #141 marker+line styles ('o-') now get the same line smoothing as '-' (2b76d4e1)
  • resample trajectory and hyp.tools.resample #94 resample= kwarg wires the existing Resample manipulator into the plot pipeline (966abe3d)
  • handling extra keyword arguments  #206 arbitrary matplotlib kwargs pass through + mismatched-length style lists now raise instead of silently degrading (715f0558)
  • use isinstance instead of is or == to test for object type compatability #209 type(x) is T → isinstance across 19+ sites, with a real crash repro (list subclass) fixed (029ed07c)
  • add tests for procrustes function with non-default parameters #199 procrustes non-default-parameter tests + better tests #244 weak-assert audit/strengthening (bf33a54f)
• 6 closed as obsolete (premise no longer applies in 1.0: memoize function doesn't consider default function args before caching #158, enhancements and fixes to DataGeometry objects and argument parsing #164, Feature request: make geo objects iterable and indexible #193, 1000 stars #204, ImportError: numpy.core.multiarray failed to import #208, matplotlib 3.0.0 bug affecting plots #220), each with an explanation.
• 22 left open as feature requests, each now carrying: current 1.0 status, a concrete implementation sketch against the 1.0 architecture, library/alternative research, an out-of-date assessment where applicable, and a new low effort/medium effort/high effort label (5 low / 9 medium / 5 high among tagged, plus imports below).

jeremymanning fork (13 issues → 0 remain)

• 7 closed as addressed with run-code evidence (incl. the 2D-GIF background and 3D-GIF camera bugs — verified at the pixel level).
• 6 migrated to ContextLab as fully-described new issues (each linking back, fork copies closed): Extend hyp.load() to support additional source types (images, obj/nii files, seaborn/sklearn named datasets, audio) #273 (extend hyp.load sources), hyp.plot(manip=...) does not exist; hyp.manip() cannot chain a list of steps -- blocks "smooth -> resample -> zscore -> UMAP" pipelines #274 (manip= plot kwarg + chainable hyp.manip — this blocks two migrated issues), "Story trajectories" demo pipeline (smooth -> resample -> zscore -> hyperalign -> UMAP -> windowed animation) cannot be reproduced -- missing manip=, animate='window', focused= #275 (story-trajectories pipeline reproducibility), Docstrings missing for hyp.manip and ~150 internal classes/methods across align/, predict/ #276 (docstring coverage), README example images/animations are unchanged since 2016-2017 (predate the 1.0 rewrite) #277 (README media refresh), Sphinx docs: hyp.manip missing from docs/api.rst; two orphaned hypertools.tools.* pages not in any toctree #278 (sphinx api.rst gaps).

Verification

• Fix-wave suite: 947 passed locally; CI status on the head commit shows on this PR's checks.
• The audit's per-issue evidence (exact commands, outputs, and rendered-image checks) is in each issue's comment; the posting log recorded 0 failed GitHub operations across the 80 issue updates.

The ContextLab tracker is now the single organized backlog for deciding what else goes into 1.0: 28 open issues, every one labeled and documented.

🤖 Generated with Claude Code