C++ search#210
Draft
ms609 wants to merge 909 commits into
Draft
Conversation
ms609
marked this pull request as draft
March 25, 2026 14:21
ms609
added a commit
that referenced
this pull request
Mar 28, 2026
ms609
added a commit
that referenced
this pull request
Mar 28, 2026
ms609
added a commit
that referenced
this pull request
May 18, 2026
In R CMD check, R runs as a non-interactive subprocess with captured stdout. R_FlushConsole() calls fflush() on that pipe; when the buffer fills the call blocks indefinitely, causing the 6 h GHA timeout seen on every ubuntu runner for PR #210. Gate the \r-overwrite progress line and the flush behind R_Interactive (FALSE in batch/check contexts). Interactive sessions are unchanged. At verbosity >= 2 in batch mode, emit plain \n-terminated lines so diagnostic logs still carry progress detail without the flush risk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ms609
added a commit
that referenced
this pull request
May 18, 2026
In R CMD check, R runs as a non-interactive subprocess with captured stdout. R_FlushConsole() calls fflush() on that pipe; when the buffer fills the call blocks indefinitely, causing the 6 h GHA timeout seen on every ubuntu runner for PR #210. Gate the \r-overwrite progress line and the flush behind R_Interactive (FALSE in batch/check contexts). Interactive sessions are unchanged. At verbosity >= 2 in batch mode, emit plain \n-terminated lines so diagnostic logs still carry progress detail without the flush risk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…te (#245) Closes the coverage gap that let the IW NNI rescore bug (3df9088) ship. The existing IW NNI test (test-ts-spr-nni-opt.R) only asserted result$score >= 0 and topology shape — it never cross-checked that the returned score matched the IW score of the returned topology. Pre-fix, nni_search did new_score = best_score + integer_delta unconditionally, so under IW the returned score was the accumulated wrong float and the existing assertions still passed. New file mirrors the EW pattern from "NNI: 15-tip random dataset" (result$score == ts_score(rt, ds)) but under concavity=10, sweeps a few concavities, and exercises multistate + NA cases. Profile parsimony isn't reached because ts_nni_search's Rcpp bridge doesn't forward infoAmounts; the IW path through compute_weighted_score is the same codepath the fix unified, so this catches both modes at the C++ level. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 19, 2026
) * line endings * feat(ls): least-squares distance scoring for the C++ search kernel Add a least-squares (LS) tree-fitting capability to the optimised C++ search kernel, so the existing NNI/SPR rearrangement engine can find the tree that best fits a target distance matrix under a least-squares criterion -- not just minimise parsimony. Built for Lapointe & Cucumel's (1997) average consensus procedure (a new sibling `Consensus` package), where an averaged, generally non-additive patristic distance matrix must be fit by a heuristic topology search. - src/ts_ls.{h,cpp}: design matrix over the tree's unrooted branches (splits + pendant edges, 2n-3 columns; root's two edges merged), OLS (normal equations + Cholesky) and NNLS (Lawson-Hanson) solvers, RSS, and NNI/SPR LS hill-climbing reusing the TreeState move primitives on a topology-only tree (never touches the Fitch state arrays). - ts_ls_fit / ts_ls_search Rcpp entry points (src/ts_rcpp.cpp, registered in TreeSearch-init.c). - R/LeastSquares.R: LeastSquaresFit() (fixed topology) and LeastSquaresTree() (search from NULL/NJ, one tree, or a multiPhylo), returning unrooted trees with fitted branch lengths and an "RSS" attr. Supports OLS/NNLS and unit/Fitch-Margoliash/custom weighting. Validated against phangorn::nnls.tree() and designTree() to machine precision (tests + dev/ls_validate.R). The existing parsimony API is unchanged. Singular (rank-deficient / zero-weight) fits fail gracefully with a warning rather than crashing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * spell: add Cucumel, Lapointe, Margoliash to WORDLIST Author names introduced in the LeastSquares documentation and bib entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
WideSample() now selects topologically spread trees by solving the Max-Min Diversity Problem with the MaxMin package, dispatching by a quality tier auto-chosen from length(trees): - exact node-packing IP (highs) for very small sets, - DropAdd-TS while the dense distance matrix is affordable, - matrix-free anchored Gonzalez (on-demand distance-column oracle) beyond that, so the largest tree sets never materialise an N x N matrix. - Imports: MaxMin (Remotes: ms609/MaxMin) instead of FurthestPoint; the GonzalezColumn / DropAddTS / ExactMaxMin calls dispatch to MaxMin. - Add Porumbel (2011) and Sayyady & Fathi (2016) references. - dev/smoke_40k.R: 40,000-tree matrix-free smoke test. - test-WideSample: 16/16 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ounded scan
Within each (has_inapp, weight) block group, order characters by descending
homoplasy (minority-state mass) as a tertiary sort key, so the bounded
indirect-length scorers reach `cutoff` in fewer blocks on rejected candidates.
Measured ~3% fewer blocks scanned on the EW reroot-x4 path (~82% of EW per-move
work).
No valid move is ever dropped: a candidate below cutoff is always fully scanned
and accepted regardless of block order; only rejected candidates bail early, and
only their discarded partial changes. The score is order-invariant (covered by
the existing "Scores are correct after block reordering" test). Because the
search is stochastic (RNG seeded from R), a faster bail can realign the stream
to a different-but-equally-valid path — reach is neutral-to-better across seeds,
not a regression.
Cost measure: `minority` = informative tips off the plurality state. Chosen over
`entropy` (log() risks ULP-level sort-key flips across platforms) and `min_steps`
(the offset-out floor; constant on multistate, =1 on all binary chars). Env knob
TS_CHAR_ORDER {none,min_steps,minority,entropy} allows retuning/disabling without
a rebuild. Full search/scoring suite green (0 failures / 1229 assertions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Durable deliverables from the TNT per-move kernel-gap investigation on project5432 (the ~24x per-candidate gap): - profiling/exactness-gate.md (+ .R harness): three-way empirical gate over 6066 candidates. Exact directional edge_set = truth; union of TRUE finals is a sound lower bound; the DEPLOYED union (uppass_node final_) strictly OVER-counts (unsafe as a screen). - red-team/union-of-finals-bound-proof.md: math-prover derivation that final(A)|final(D) subset E(D) for the deployed finals => upper bound, the reverse of the old in-code comment. Grounds the ts_fitch.cpp comment fix. - profiling/l3b-footprint-482.md: L3b changed-view footprint collapses with N (mean fp_ref 0.18 at 482t) -> incremental edge-set FEASIBLE at scale; refutes the small-N "dead" verdict. Feasibility basis for lever #6. - profiling/s7-fastpath-sizing.md: #7 sized to ~5% whole-search (not 10x); the 64-bit Hamilton TNT is SSE2 (not scalar) -> the gap is fewer-words-per-move, not per-word arithmetic. - plans/2026-07-14-lever7-scorer-monomorphization.md and plans/2026-07-14-lever6-incremental-edgeset-land.md: self-contained build specs (land #7 first). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The NOTE on fitch_indirect_length claimed the union final(A)|final(D) is a SUPERSET of the true directional edge set and therefore UNDER-counts (a sound lower bound). That is backwards for the finals the function actually reads: tree.final_ is the simplified uppass_node up-pass (final subset prelim, never unions the parent state in), so final(A)|final(D) subset edge_set[D] = MPR(D) => the union OVER-counts and is an UPPER bound, NOT a sound lower bound. Documentation only; no behavioural change. The function stays a heuristic ranker (temper) whose chosen move is re-scored exactly. Comment now warns it must NOT be used as a pruning/rejection screen (it can discard improving moves) and cites the proof at dev/red-team/union-of-finals-bound-proof.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lift the per-candidate criterion-flavour dispatch out of the plain-EW SPR hot loop into a compile-time template (spr_scan_plain_ew<UseFlat,Wrong>), selected once per weight-class at the scan site. The dead-in-plain-EW branches (NA/IW/sector/constraint/collapsed/b2) compile away, and the UseFlat=true instantiation uses the flat kernel (fitch_indirect_cached_flat) — weight-blind, so UNSAFE as a global toggle but SAFE as an instantiation because dispatch selects it only for all_weight_one data. EXACT / byte-identical: default ON, kill-switch TS_EW_MONO_OFF. Gate passed on Wortley2006/Zanol2014/Zhu2013 x>=2 seeds via ts_tbr_diagnostics AND MaximizeParsimony (score + candidates_evaluated + per-pass identical), plus a clean-HEAD binary cross-check and a positive control (split flat/cached fire counters + a diverging wrong instantiation). The MaximizeParsimony gate caught + fixed a real bug: collapsed_all_zero went stale across mid-search compute_collapsed_flags recomputes; now refreshed at every recompute. Per-candidate SPR: +12-19% on all_weight_one (flat kernel), +6% on weighted (dead-branch strip). Whole-search ~2-5% unit-weight / ~1% weighted, exact. Orthogonal to 5432 reach. See dev/profiling/s7-land.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…easure Address review before hand-off: - Compile-time-gate the TS_EW_MONO_WRONG corruption behind -DTS_EW_MONO_WRONG (like the removed TS_EW_MONO_ASSERT probe). A production binary now has NO env-var path to a corrupted scorer; verified TS_EW_MONO_WRONG=1 is INERT on the default build, and the positive-control divergence reproduces only on a flag-built binary. Closes the footgun. - Widen the MaximizeParsimony byte-identity gate to 6 seeds x 3 datasets (18/18 PASS) confirming the collapsed_all_zero-refresh fix is not seed-luck. - Add the faithful whole-search wall-clock (full MaximizeParsimony, nThreads=1, min-of-runs): Zhu2013 +1.3%, Zanol2014 +0.1% — lower than the single-descent proxy (+2.7%) because ratchet/sector/enum phases don't hit the flat SPR path. Reword the cross-dataset flat-vs-strip comparison. Final airtight preclean (CCACHE_DISABLE) build re-confirms byte-identity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eteness Post-review doc polish: state the whole-search number as the one resolvable unit-weight point (Zhu +1.3%) rather than firming a range; record that collapsed_all_zero is the ONLY per-clip-mutable input to the ew_mono_plain gate, so refresh-at-every-recompute closes the whole staleness class (not just the Zhu2013/seed202 instance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add spr_scan_plain_iw<Wrong> and a 3-way SPR dispatch (EW-flat / IW / general loop) gated on a shared refreshed mono_plain predicate. IW has no flat kernel (implied weights are per-pattern), so the win is the dead-branch strip only; the scorer + accept are reproduced verbatim from the general loop's IW branch. Byte-identical (default ON; TS_EW_MONO_OFF reverts both EW+IW): - ts_tbr_diagnostics IW 6/6 + MaximizeParsimony/XPIWE 9/9 (score to full precision, candidates_evaluated, per-pass all identical); IW template fired 100% of clips (iw fire counter); wrong IW instantiation diverges on the -DTS_EW_MONO_WRONG build. EW regression re-checked 6/6 after the refactor. - Per-candidate IW SPR-specific +2.7-4.0% (strip only; the heavier IW double-accumulator scorer dilutes the fixed branch fraction below EW's ~5-6%). IW/XPIWE is the production default, so this is the common path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mpleteness) Independent supervisor review found the lever-7 collapsed_all_zero gate could go stale at ONE recompute site the "every recompute" fix missed: the legacy physical-reroot path (TS_PHYS_REROOT=1, ~ts_tbr.cpp:2756) recomputed `collapsed` without the refresh_collapsed_all_zero() that the default fast-reroot (2735) and accepted-move (2671) sites carry. Latent (default phys_reroot=false never runs it; NEW==BASE stayed byte-identical on Zhu2013 101/202 even under TS_PHYS_REROOT=1), but it broke the pattern and could diverge where a physical reroot flips a non-root-child zero-length branch into existence. Add the same refresh at 2756. Re-verified byte-identical NEW-vs-BASE (via the TS_EW_MONO_OFF kill-switch) on the default path (ts_tbr_diagnostics 6/6 + MaximizeParsimony Zhu2013 101/202, cand identical) AND under TS_PHYS_REROOT=1 (1.08-1.22B candidates identical). s7-land.md updated to note the third site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Objective (i) always-find-best-score and (ii) be-as-fast-as-possible are distinct problems on the current frontier: - Mission A (cold-start basin capture): 5432's 3-step/0.15% gap is PURE cold-start basin capture (1943 is a stable TS optimum TS's TBR rebuilds from within ~8-12 TBR; cold search lands ~109 away). Lists every refuted lever, the one that moved the floor (reweighting kick 5->auto), and the open constructor angle. Existence-before-build. - Mission B (8x kernel): the ~30x per-move gap; the live lever is the Goloboff 1996 up-aware approximate screen. Gated on read-the-paper + vary-n_states discriminator + naming where TS is kernel-bound-and-losing, and a sound-bound soundness proof (the up-ignoring screen was proven unsound). Self-contained briefs so each mission can run in its own fresh session off auto-memory. #6/#7 (per-move micro-levers) landed; these target the real gaps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rift/sector opt-in + findings (#273) * docs(profiling): settle union-of-finals exactness gate (empirical) Fresh rooted union-of-finals is NOT an exact EW-Fitch insertion cost, under any rooting or on binary data. Three-way empirical verdict over 6066 reconnection candidates vs TreeLength full-rescore ground truth: - Per-child edge set E(D)=combine(prelim(D),up(D)): EXACT (6066/6066). - Union of TRUE edge sets E(A)|E(D): sound LOWER bound (under-counts, never exact). - Union of the DEPLOYED simplified final_ (fitch_indirect_length): OVER-counts (unsafe as a lower bound), because uppass_node's simplified rule never unions so final_ is a strict subset of prelim. Fresh finals => formula limitation, not staleness. The code comment claiming it under-counts is wrong for the deployed path. => "root like TNT -> cheap exact union-of-finals" refactor is a mirage. Adds dev/profiling/exactness-gate.md (verdict) + exactness-gate.R (harness). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(drift,spr): opt-in exact directional EW scorer + scan-error census The DRIFT phase (ts_drift.cpp) and standalone spr_search (ts_search.cpp) still scored pure-EW reconnection candidates with the union-of-finals approximation (fitch_indirect_length_bounded, reading the incrementally-maintained tree.final_), never migrated to the exact directional edge set that main TBR adopted in 2b299e4. Severity (dev/profiling/drift-exactness-gate.md): no inexact score is ever leaked (every accept is drift_full_rescore'd / full_rescore-verified), but on the deployed incremental path the union error is TWO-SIDED (over- AND under-counts up to ~100 steps, not the strict over-count the exactness gate measured with fresh finals). argmin selection then suffers an optimizer's curse (picks moves the scan is optimistic about), so drift admits moves outside its AFD envelope (env_violation 52-62% on Zhu2013/Zanol2014) and spr_search converges prematurely (union 810 vs exact 639). Fix (opt-in, union stays default; NA/IW untouched, gate !has_na && !use_iw): - TS_DRIFT_EXACT routes drift's SPR + reroot EW scans to fitch_indirect_length_cached over a per-clip compute_insertion_edge_sets, mirroring tbr_search. - TS_SPR_EXACT does the same for spr_search. - TS_DRIFT_SCANCHK: two-sided scan-error + decision census (select_flip, env_violation, applied-move predicted-vs-full_rescore; cf. TS_IW_SCANCHK). Validation: exact path applied_mismatch == 0 across all applied moves incl. reroots; default (flags-off) path byte-identical (drift/SPR/char-ordering tests pass). Local matched-wall pilot: frontier CROSSOVER -- exact wins the auto/ wall-race regime on all 3, union wins the thorough/saturated regime on 2/3 -> land opt-in, do not flip the default. spr_search is an unambiguous reach win. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(hamilton): drift-exactness matched-wall gate build + array scripts Dedicated build (lib-driftexact, does not clobber the shared cpp-search lib) fetching claude/happy-jennings-e7a165, and a 3-task array (one per dataset) running dev/profiling/drift-exactness-gate-bench.R at 30 seeds x nCycles {8,16,32,64,128,256}, union vs TS_DRIFT_EXACT, to resolve the matched-wall plateau-vs-climb crossover the local pilot showed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(hamilton): resolve gate deps from shared lib (R_LIBS_USER) lib-driftexact holds only the branch's TreeSearch; point R_LIBS_USER at the shared TreeSearch/lib so Rcpp/TreeTools/ape resolve at build+run time while the harness loads our TreeSearch via lib.loc=GATE_LIB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): close matched-wall gate with 30-seed Hamilton result Hamilton array (17870087, 30 seeds x nCycles {8..256}, 3 datasets) confirms the 8-seed pilot's matched-wall CROSSOVER with clean monotone gaps: exact wins the auto/wall-race regime on all three (~2.7-3x faster, no time-to-optimum regression), union wins the thorough/saturated regime on 2/3 (Zhu2013, Zanol2014 - exact plateaus, union over-diversifies deeper); Dikow2009 exact dominates every budget. best-of-30 equal on all three (reliability effect, not reachability). Verdict: LAND OPT-IN (TS_DRIFT_EXACT), do not flip the drift default. Adds the per-dataset raw CSVs and the frontier reanalysis script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): soften saturated-tail claim (nCycles-cap censoring) The nc<=256 gate compared a censored exact tail (exact ~3x faster, so frozen at its nc256 score with unspent wall) against union's fuller curve; exact was still descending nc128->256, not plateaued. So "union wins thorough on 2/3" is not established. auto/wall-race win stands. Extension (exact+union nc {512,1024} on Zhu2013+Zanol2014) running to settle at matched wall. Opt-in landing unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): resolve saturated tail — crossover is real (nc1024 check) Extended exact+union to nc {512,1024} on Zhu2013/Zanol2014 (30 seeds) to remove the nCycles-cap censoring. Exact does NOT plateau (Zhu 624.80->624.40) but descends slower than union; at genuinely matched wall union stays ahead through the saturated tail (Zhu floor 624.0 vs exact ~624.4; Zanol narrower ~0.13). So the auto/thorough crossover is real, not an artifact: exact wins the wall-race regime on all three, union wins the thorough regime on 2/3. Verdict CLOSED: land TS_DRIFT_EXACT opt-in, do not flip the drift default. Adds the extension CSVs and the combined-frontier reanalysis script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): sector-godrift isolation gate harness + Hamilton array (route B) Full "thorough" search with driftCycles=0 (main-tree drift off; verified gd=0 => 0 drift calls, gd=25 => all-sectoral) isolates the in-sector godrift scorer. Three arms none/union/exact, maxReplicates knob, matched-wall frontier. Array adds mbank_X30754 (180t) as the larger/harder case where sector drift is most active. Reuses lib-driftexact (no rebuild). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(spr): make exact directional the default EW scorer for spr_search spr_search is a pure hill-climb with no exact-TBR phase to launder union's over-count, so union converges catastrophically short (multistart best-of-25: Zhu2013 650 vs exact 625; Zanol2014 1292 vs 1265 — systematic, not variance), and exact is also faster per start. No crossover, unlike drift. Flip default to exact; TS_SPR_UNION restores the old scorer (kill switch). Tests green (SPR/spr-nni-opt/spr-state-restore/drift). Reach today = standalone ts_spr_search (sprFirst OFF in presets), but this makes an SPR warmup viable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): route-3 marginal-value experiment (drift vs ratchet at matched wall) From a shared per-seed local optimum, compare where a wall-second is best spent: incumbent ratchet vs exact-drift vs union-drift (control). If exact-drift's anytime frontier beats ratchet's over some wall range and dominates union-drift, drift earns a place in wall-limited recipes (route 3) -> worth re-tuning. mbank_X30754 (180t) is the large/hard anchor. Reuses lib-driftexact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bench): route-3 follow-ons — +replicate gap-closer + training-corpus recipe test arm A (+replicate): is a wall-second better on exact-drift than on the project's real incumbent lever, more RAS+TBR replicates? Same datasets as route3 for frontier comparison. arm B' (recipe): whole-search thorough on TRAINING-split MorphoBank (EW-recoded, one key/task, sector drift off), arms none/union/exact main-tree drift, to see if the drift scorer choice moves a wall-limited recipe. Both reuse lib-driftexact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(drift): route-3 recipe verdict — NO-GO (marginal win, ensemble-redundant) Marginal value (from a shared local-opt): exact-drift is the best single continuation, beating ratchet + union-drift + replicate. But the whole-search recipe test on 8 training-split MorphoBank keys (EW, sector-drift off, sequestered) shows main-tree drift is REDUNDANT in the multistart+ratchet+ sectorial+TBR ensemble at both saturated (none ties + faster) and tight-budget ends (exact beats none only 1/4 keys). So the isolated win doesn't survive -> no recipe surgery; exact-drift stays opt-in (TS_DRIFT_EXACT), D preset-refactor not worth it. Adds companion note + raw CSVs + analysis. (spr_search exact default stands as the real landed win from this line.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… Scheme 1) Exact incremental replacement for the per-clip from-scratch compute_insertion_edge_sets recompute, gated to the plain directional EW/IW TBR regime (TS_L3B_INCREMENTAL, default OFF). Patch-from-full "Scheme 1": maintain a pristine per-pass intact-tree base; per clip patch only the changed frontier (worklist incremental up-pass, value-based attenuation stop), score, then restore base over the changed-node list. Shares ts_fitch_combine with the from-scratch path so patched words are byte-identical. Chose Scheme 1 over the design doc's Euler "Config C": the 482t evidence that reopened the lever (fp_ref) supports Scheme 1; the Euler descend-delta was killed at small N and never re-measured; the clip loop already restores per clip (Scheme-1-native) and keeps the shuffle => bit-identical C-vs-A. Advisor concurred. Rationale + results in dev/profiling/l3b-land.md. Correctness: per-clip oracle (edge_set == from-scratch, full array) mismatch=0 across random EW (64 configs), real multi-state/ambiguous panel + project5432 subsamples, EW+IW, single-rooting and do_reroot; C-vs-A bit-identical; tbr completeness oracles clean. do_reroot supported (not a footprint penalty). Wall (raw tbr_search, real project5432, bit-identical trajectory): wash at <=75t, 1.10x@120t, 1.39x@240t, 1.36-1.38x@482t (both unrooted) — the predicted large-N ceiling. Open: end-to-end MaximizeParsimony at shipped ratchet on Hamilton (sector share is gated out, so end-to-end is diluted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full MaximizeParsimony C-vs-A on project5432 (EW): 1.01x wash at both 120t and 240t, despite raw tbr_search being 1.39x at 240t. Coverage confirmed (l3b fires in driven+ratchet; sector ~30% gated out) and the per-clip oracle is clean in the ratchet upweight/collapsed context. The raw-TBR win is diluted to nothing end-to-end by the sector exclusion + per-pass base-recompute amortization across ratchet's short passes. Verdict: exact/correct large-N raw-TBR lever, NOT a net mission-wall lever as built -> land default-OFF; the unlock is incremental base update after accept. Replaces the earlier ~1.2x end-to-end estimate with measured numbers; marks the Hamilton 482t full-search run not-worth-running until the base-update follow-on lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ran the cheap discriminator (MP 240t, TS_L3B_STATS+ORACLE): in-search fp_changed = 0.244, essentially the clean 240t footprint (0.243) -> the per-clip patch win is INTACT, refuting "in-search fp elevated -> win structurally small". So the wash is sector-exclusion (predicts ~1.24x) + the per-pass O(n_node) base recompute, which MP's short ratchet passes amortize poorly. Confirms base-incremental-update as the plausible unlock; notes it is itself a correctness-critical kernel (TBR-accept, two dirty paths + reroot, harder than the clip patch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… honest verdict Replaces the per-pass full compute_insertion_edge_sets base recompute with an O(footprint) in-place update after each accepted SPR move (update_base_after_spr_move): the move decomposes into a remove-leg at nz + a single-site insert-leg at above; nx gets a fresh up so A1-path up is recomputed (not assumed invariant). Gated SPR-only (TBR-reroot/reroot/first-pass fall back to full recompute); pass-top recompute skipped when l3b_base_current; buffers synced over the changed set; kill switch TS_L3B_NOBASEINCR. Base oracle (base == from-scratch after each SPR accept) clean across EW+IW tbr_oracle (to 120t x18) and full ratchet MaximizeParsimony on real Zhu2013/Zanol2014; first-version root-child seed bug caught by the oracle and fixed. Measured (project5432, EW): base-incr removes the per-pass recompute, converting plain-l3b's small MP regression (0.99x @240t) into a clean wash (1.00x), and preserves raw tbr_search 1.39x@482t / 0.97x@37t. But it yields NO net MP win => the base recompute was NOT the bottleneck. The MP wash is structural: short ratchet-recovery passes (few clips/pass) + sector ~30% excluded. Verdict unchanged and now complete: exact large-N raw-TBR lever, not a mission-wall lever; land default-OFF. STOP (advisor gate: fixed cost removed, still no win). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The L3b incremental edge-set patch is EXACT (bit-identical trajectory) and 1.36-1.38x faster on raw large-N TBR (dev/profiling/l3b-land.md); end-to-end MaximizeParsimony is a clean wash (neutral, with the base-incremental follow-on). Per objective (ii) be-as-fast-as-possible, take the free large-N win: default-ON for n_tip>=150; small trees (where the raw-TBR win washes out) pay zero overhead. Gate: l3b_active = !TS_L3B_OFF && (TS_L3B_INCREMENTAL || n_tip>=150) && use_directional && ... TS_L3B_OFF kill-switch (full disable) TS_L3B_INCREMENTAL force-on regardless of size (A/B + small-tree testing) Verified on the merged+flipped build (CCACHE_DISABLE --preclean; header changed): force-ON == force-OFF byte-identical on the panel (Zhu/Zanol x2 seeds: score+n_evaluated+passes), l3b fires (patch_clips=3909), n_tip<150 default inert (== pre-#6 #7 baseline). The n_tip>=150 exact path was validated at 482t by the chip's per-clip oracle + C-vs-A. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-seed assertion (seed 1001) that zero-mode and upweight-mode ratchet produce different search statistics coincidentally held equal on macOS after d8c5999 (homoplasy character ordering) shifted the RNG-seeded trajectory under libc++'s std::sort tie-break. On a 15x10 binary matrix the two modes coincide on ~3-7% of seeds; which seeds coincide is platform dependent, so a single fixed seed is fragile. Assert instead that the modes diverge on >=1 of several seeds (early break), so an incidental single-seed coincidence no longer fails while a genuine collapse (upweight silently == zero) still does. The deterministic regime-distinctness guarantee is already covered by test-ts-na-evcache.R. Validated green on a clean committed-HEAD build (CCACHE_DISABLE=1 --preclean): all 9 blocks pass, no crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…liverable ratchetPerturbMaxMoves 5->auto/scaled corpus anytime A/B (Hamilton job 17872154, 25 training matrices x 3 arms, validation sequestered) => NO broad wall benefit: 14/25 tie exactly; clean large/xlarge reach the optimum in the same replicate (rep2hit ratio 1.000); narrow reach regression on a degenerate 4062t giant; marginal noisy +reach on one hard multi-basin matrix. Verdict: keep the default at 5; the kick is an opt-in hard-tail lever, not a default. Rewrite dev/plans/2026-07-11-...: retire the FALSIFIED clade-generation thesis (restore238 shows TS's TBR rebuilds 1943 from within ~8-12 TBR moves), reframe to cold-start basin capture, correct the numbers, fold in the kick result. Adds kick_anytime.R (harness), kick_anytime_hamilton.sh (SLURM array), kick_analyze.R (analysis), kick_anytime_FINDINGS.md (write-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…from 1943) Mission A angle #2 (push the reweighting kick as a basin-HOPPING schedule to cross the last steps past the ~1945 TS floor on project5432). Mechanism: src/ts_ratchet.cpp is a MONOTONE strict-descent ILS (accept-only-if- strictly-better, else reset-to-best), so strength/schedule/multi-kick all stay inside a monotone frame; the only escape lever is the acceptance frame (accept-worse: drift, or reweighting with accept-worse between cycles). Gate (buildless, dev/benchmarks/basin_hop_gate.R): ordered-char re-score by label (self-check L(1943 floor)==1943) + TBRDist(exact=FALSE, hard lower bounds) to the 1943 floor. Every near-optimal TS tree from EVERY generation method (kick 1945 / strongperturb / cold / random / largefirst / invweight, 1945-1958) is tbr_min >=97 TBR from 1943 (min 97, best kick only 2 score-steps above 1943). Corroborated by basin_5432 (descent 0/5 by d~20) and geom_full (~109). No accept-worse schedule walks ~100 rearrangements to a narrow distant basin, so the afd-sweep probe and the accept-worse build were NOT run. Scope: refutes route #2 only (escape the floor). Angle #1 (cold-start constructor) and #3 (structured breadth) are untouched - they generate a start rather than walk from a floor. Kick is not useless (CID 0.22 vs cold 0.38: a better LOCAL basin). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gating [focus-test: ts-ratchet-opt] R-CMD-check gates the macOS `full` job behind the ubuntu/windows `core` wave (full needs core needs sense-check), so a flake elsewhere stops the macOS run from ever building. focus-test runs ONE testthat file on ONE OS (default macOS-latest) with no full R CMD check -- via the Actions tab (workflow_dispatch) or a [focus-test] / [focus-test: <filter>] marker in a commit message. This commit's own [focus-test: ts-ratchet-opt] marker triggers a macOS run of the de-flaked ratchet test as a live check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto-memory is now organised as campaign hubs indexed at the top of MEMORY.md; record the write-side rule (advance a campaign by updating its hub the same turn) on the always-loaded project-instructions channel so future agents follow it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o diameter build)
Zero-code precondition for a TNT-style dss (diameter-limited local sector) build.
The dss benefit (local sector -> cheaper escape) is driven by TIP-COUNT, which the
existing sectorMaxSize knob controls; rev5's missiongate sweep never varied it small
(held maxS>=0.65n). This probe sweeps small maxS {12,20,30} x ras=3 from the fixed
TNT T0, vs a large-sector anchor and ratchet, on the missiongate protocol.
Local pilot (Zanol s1, Zhu s1, n=3) + Zanol reach test (n=16 restarts, up to r=256):
- steps/Mcand APPEARS to favour small sectors (-2.4 vs ratchet -1.5) but this is a
candidate-COUNT artifact: small-sector candidates cost ~2x ratchet's per wall
(per-pick reduced-dataset build + 3x RAS overhead), so on the WALL axis (the true
mission axis) ratchet dominates the hard target (-29 vs -24 steps/s, reaches deeper).
- REACH CEILING: small sectors PLATEAU at -12 on Zanol at ANY budget; ratchet reaches
the -13 target. Whole-tree-barrier prior (rev2): a sector <= maxS tips cannot emit
the split(s) for the -13 reorganization.
=> dss/diameter-limiting is strictly MORE local -> shallower reach -> cannot help.
No C++ build. Confirms + extends rev5 (sectorial Pareto-dominated by ratchet) to the
small-maxS regime it never measured. Hamilton n=15 confirmation scripts included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-a source dir gone on Hamilton) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…losed Hamilton array 17898332 (build 17898331, HEAD 023b038), 12 cells x n=15 (3 TNT-T0 seeds x 5 reps), corrected wall+reach aggregate. VERDICT 0/4: no small-sector config reaches ratchet's median depth at <= its wall. On the hard Zanol target ratchet dominates EVERY axis incl the biased steps/Mcand (-1.90 vs best smallmax -1.78). The n=3 local-pilot "lead" (maxS=12 r=1 = -2.39 st/Mcand; a Zhu wall win) was noise + metric bias: at n=15 it falls to -1.19 and the Zhu win vanishes (ratchet med -3 vs smallmax -1). steps/Mcand is a BIASED axis here: small-sector candidates cost ~2x ratchet's per wall (reduced-dataset build + 3x RAS overhead over fewer candidates), so it flatters small sectors. Adjudicated on WALL + reach instead. => dss/diameter-limiting is strictly MORE local -> shallower/less-competitive -> cannot help. No C++ build. Confirms + EXTENDS rev5 (sectorial Pareto-dominated by ratchet) to the small-maxS regime rev5 never measured. Also refutes rev5's handoff nuance (small maxS + ras=3 vs single ratchet start on Zhu/Giles): smallmax median <= ratchet median on all cells. Full writeup: DSS_FINDINGS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… retested The selectem_large anchor used auto picks (~3/round), under-provisioning rev5's selectem (explicit pk 5-20): my Zhu selectem_large median -1 vs rev5's -6. So it is NOT a faithful rev5 reproduction and cannot speak to rev5's handoff (large-band selectem being lower-VARIANCE than a single ratchet start on Zhu/Giles — a variance question never measured here). Retract "refutes rev5's handoff"; that stays OPEN as rev5 left it. Core verdict UNAFFECTED: the smallmax arm used auto picks too, but for small bands that gives MANY picks (~16/round at maxS=12) = over-provisioned, so its 0/4 loss vs ratchet (esc_med-vs-ratchet-at-wall, same pooled seeds) is decisive. Also de-emphasised the cross-seed-pooled reach flag as non-load-bearing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TS_PACK_LOCAL packs each 64-char block to its own applicable-state alphabet
rather than the global one, cutting Fitch planes scanned per candidate on
low-state morphological data (~1.1x small -> ~1.5x char-rich, climbing with
nChar). Score-EXACT per tree; now default-ON (set TS_PACK_LOCAL=0 to force off).
- ts_data.cpp/.h: default-ON flip; add CharBlock::plane_state[] mapping each
packed plane back to its global applicable label (identity in the unpacked
layout), populated in build_dataset by inverting loc[] through state_remap.
- ts_rcpp.cpp: ts_na_debug_char reconstruction consumes plane_state[] so packed
output prints the global state label, not the plane index.
- test-ts-pack-local.R (new): score-exactness (TreeLength ON vs OFF) + a
hand-built non-contiguous-alphabet fixture that exercises plane_state[] -- the
only gate that does, since the scorer is index-agnostic.
- reeval pack_{gate,reach,mpt_gate}.R: OFF-arm switched to TS_PACK_LOCAL=0
(unset now means ON).
Validation: pack_mpt_gate Sansom2010 EW/IW/NA byte-identical ON vs OFF (score +
ntree + candidates_evaluated, 3 seeds; NA reproduced the 66-tree MPT set); full
testthat default-ON 0 failures (11337 pass); corpus 18/18 reach-neutral.
ts_na_debug_char is package-internal (no exported surface changed) and
PlotCharacter is pure-R, so no user-visible reconstruction changes. The packed
ambiguous-set content differs (local vs global alphabet) -- correct, not a
regression: phantom states never appear in an MPR, proven by identical
score+candidates_evaluated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give the 4-wide flat reroot batch kernels (fitch_indirect_cached_flat_x4 / indirect_iw_cached_flat_x4 + NA) the same next-row software prefetch the scalar reroot path already has. The x4 batches issued 4 cold demand loads to scattered vroot_cache rows per batch with no hint; the in-batch MLP can't hide the batch-boundary head latency (cutoff early-break is a control dependency). EW x4 prefetches kept_ei[ki..ki+3] (next batch). IW x4 tracks a parallel iw_pf cursor into kept_ei -- valid because its skip predicate is identical to the kept_ei construction (use_collapsed == !collapsed.empty(), ts_tbr.cpp:1532), so kept_ei is the same candidate sequence in the same order. Purely additive and flag-gated (opt-in TS_TBR_PREFETCH=1), so flag-OFF is byte-identical to prior committed code by construction; a prefetch hint cannot change a score anyway. Byte-identity validated: ts_tbr_diagnostics ON vs OFF on project970, EW + IW, identical score + n_candidates_evaluated (38.9M/44.8M) + n_clips + final edge. Local Windows/MinGW A/B flat (project970 60.0->60.2, project510 57.1->57.0, no regression) -- expected, since the memory-latency win only surfaces in the EPYC L3-pressure regime. Landed default-OFF as a validated building block per the lever-6 precedent; the win/wash is settled on Hamilton EPYC raw-TBR + full MaximizeParsimony wall before any default flip. See dev/profiling/kernel-prefetch-x4.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the +1 slack on the EW/NA integer bail cutoff (best_candidate - divided_length + 1 -> + 0) at the EW sites: spr_scan_plain_ew template, SPR general, reroot x4, reroot scalar. The scorers bail on extra_steps >= cutoff; the +1 let an equal-length, non-improving candidate accumulate its full extra before rejection (scanning every block on the near-optimal plateau), whereas +0 bails it the moment its running extra reaches the current minimum. Byte-identical: acceptance is strict < everywhere, so a bailed candidate (extra >= cutoff = best-divided) is rejected identically, a true improver never reaches the tighter cutoff, and an equal candidate bails one block earlier but is still rejected. The +1 was not protecting acceptEqual (decided post-scan on the recorded best_candidate), and the drift path already ships slack 0. Gated by a cached tbr_cutoff_slack() (default 0; TS_TBR_LOOSE_CUTOFF=1 reverts to the historical +1 for the A/B). Left the +1 on spr_scan_plain_iw (cutoff dead on the IW path -- IW bounds on the float best_candidate), the IW-float cutoff, and the INT_MAX init. Validated byte-identical (score + n_candidates_evaluated + n_clips + final edge) tight vs loose on project970 EW + IW + Sansom2010 NA. Chosen by the post-packing lever-hunt workflow with 3/3 adversarial verifiers confirming exactness + non-wash. Local ns A/B is flat (blocks-scanned is hardware-independent, so representative): the exact-equal-full-scan population is small in practice (dedup skips equivalent rerootings; worse candidates already bail early). Landed default-on regardless -- byte-identical, guaranteed non-negative, the correct tighter bound; may pay in the raw-TBR @482t plateau regime, to be checked on the EPYC A/B. See dev/profiling/kernel-cutoff-tighten.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ened it) Whole-program -mavx2 is 8-9% faster per candidate than the package -msse2 build (project970 60.6->55.2, project2668 34.7->31.9) because the target(avx2) reduce helper is a non-inlined call boundary and packing (ns 2-4) made that a big fraction of the block body. Pre-packing this was -1.2% (T-P5f); packing reopened it. The portable source capture (target(avx2) scorer clone + cpu_has_avx2 dispatch) was flat locally and reverted: cpu_has_avx2() is almost certainly false on this MinGW/Windows box (__builtin_cpu_supports unreliable there), so the shipped Windows binary likely runs SSE2 -- a separate portable win. On Linux/EPYC the runtime check works, so the baseline already uses the AVX2 helpers and the clone (boundary kill) is the right lever there. Resolution deferred to a Hamilton EPYC -msse2-vs-mavx2 A/B (after the 5432 packbuild job frees the lib). See dev/profiling/kernel-avx2-reduce-gate.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0% end-to-end EPYC A/B (job 17908514): the shipped -msse2 build pays a per-block AVX2-call boundary (cpu_has_avx2 true on Linux, but the target(avx2) helper can't inline into the -msse2 scorer). -mavx2 removes it: 1.67x/1.38x/1.26x per candidate on project970/510/2668, scaling with block count. Windows masked this (cpu_has_avx2 false there -> inline SSE2 -> gap looked like 9%). But end-to-end (job 17908602, fixed-seed maxReplicates=4, identical trajectory): project970 1.01x, project510 1.10x. Amdahl: scoring is only ~3% (970) / ~34% (510) of MaximizeParsimony wall; the rest is non-scoring machinery (ratchet/ sectorial/tree-ops). The per-candidate win does not propagate -- same dilution that sank lever-6, and why packing (shrinks total_words pervasively) beat AVX2 (reduce compute only). Banked as a -march=native Hamilton deployment build (dev/profiling/tsmarch_build.sh -> /nobackup/pjjg18/tsmarch/lib): free ~1-10% for EPYC runs, machine-specific, NOT for CRAN (breaks non-AVX2 CPUs; release keeps -msse2 + runtime dispatch). The portable target(avx2) clone is not worth it (small, dataset-dependent end-to-end). Confirms the mission: time-to-optimum is bounded by search machinery/stopping, not the per-move kernel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t=off Verified the -march=native deployment lib gives byte-identical EW+IW scores vs -msse2 (project970, 8 trees, IW k=10, 12dp). Add -ffp-contract=off so byte-identity is guaranteed (not just empirical): keeps the AVX2-inline win (boundary-kill, not FMA) but forbids FMA fusion of the IW/profile float accumulators, which could otherwise shift IW ties -> trajectory drift in a race. tsmarch/lib is thus a transparent speedup, safe for IW-tie-sensitive race runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual testing underway; shiny app in particular has some usability issues.