Implements new tile57 API#45
Merged
Merged
Conversation
POST /api/debug/partition bakes an ownership-partition PMTiles for every installed
provider (which cell renders which ground per band, no portrayed content) via the new
tile57.BakePartitionDebug binding, and registers each as the "{provider}-partition" tile
set — served at /tiles/{provider}-partition/{z}/{x}/{y}.mvt (layers: "partition" polygons
by cell colour, "labels" points with the owning cell name). GET returns each provider's
readiness + tile URL, so a debug UI can trigger generation and overlay the result.
Reuses encRootDir + setDir + the tileSets registry; the partition math is all in the
engine. Verified live against the NOAA provider (3528 cells, z0-12): bake ~12s, tiles
serve 200 with both layers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Settings → Advanced gains an "Ownership partition (debug)" section:
- "Generate partition overlays" POSTs /api/debug/partition (server bakes one partition
PMTiles per installed provider), then polls until each is ready.
- Per provider, a "Show/Hide overlay" toggle adds/removes a MapLibre vector source over
the chart from /tiles/{provider}-partition: a fill layer coloured by the face "color"
property + a symbol layer labelling each face with its owning "cell". Re-applied on
style rebuild (mariner change); removed on teardown.
Lets you see the composite ownership quilt directly on the chart to debug which cell
renders which ground per band. JS syntax-checked; server side verified live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bakeProvider now bakes each cell to its own native-scale PMTiles (coverage embedded) and streams them through the engine's ownership partition into tiles/chart.pmtiles (tile57.BakeCell -> ComposeFiles), replacing tile57.BakeBundle's in-bake cross-cell combiner. Only tiles/chart.pmtiles is needed by the serving path — the MapLibre style is built dynamically (tile57.Style) and the sprite/glyphs/colortables are global server assets — so the composite bake skips the bundle's (unused) assets/style/ manifest emission. Progress is now per-cell (bake i/N) then a compose phase, driven from Go. Escape hatch: TILE57_LEGACY_BAKE=1 restores the BakeBundle path while the composite model beds in. KNOWN GAPS to flush out: per-cell bake is serial (a warmup + worker pool is the next lever for whole-district speed); the compose phase has no fine-grained progress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…drop the tree) composeProvider swallowed per-cell BakeCell errors and returned (0, nil) when every cell failed, which bakeProvider treats as "no coverage" → os.RemoveAll(outDir). In the colocated cache+data test layout that deletes the provider's ENC_ROOT source. BakeBundle errors in this case (no removeAll), so match it: return an error when cells were present but none baked. Fixes TestImportFetchDownloadOnly on the compose path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…shared helper) Extract the compose flow (walk cells -> BakeCell each -> ComposeFiles) into baker.ComposeENCRoot, shared by the server bakeProvider and the CLI runTile57Archive. Both now default to the ownership-partition composite; TILE57_LEGACY_BAKE=1 restores the in-bake BakeBundle combiner. The CLI reads the manifest bbox back from the composed archive (OpenPMTiles.Info). Server composeProvider is now a thin wrapper that maps the onProgress/onSkip callbacks to the import job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The composite provider bake now reports seconds-remaining (mean per-cell rate over cells baked so far) on the import job, and the CLI prints it too. The web progress UI ignored the job's free-text `note`, so the earlier note-only ETA never showed; add a dedicated `eta` field to the import job and render it in chart-service._formatStatus via a compact fmtEta → "1,234 / 4,567 charts · ~2m left". Rebuild + hard-refresh to pick up the embedded web asset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The composite bake's 3-5 min partition compose was one opaque step: the UI stuck on "Preparing" (the client re-derived the verb from phase+unit and pinned bake+cells -> "Preparing") behind a frozen 100% bar. Now: - composeProvider feeds tile57's new ComposeFiles progress callback into the job: "Composing tiles" with a real bar (engine zoom-weight fraction), a "zoom N of M" detail, and an ETA extrapolated from elapsed compose time. The per-cell stage reports "Baking charts". - The server owns the status wording: statusJSON emits display-ready action + detail + frac (bar fill; null = indeterminate) built server-side; the client renders them verbatim instead of interpreting phase/unit/note. The dead client PHASE map / fmtBytes / fmtEta and the "Preparing" heuristic are removed; the raw fields still ride along for pack/region context. - The earlier note-only ETA now actually reaches the client (statusJSON had dropped the eta field, so the wired-up fmtEta path was dead). Rebuild with `make TILE57=../tile57 build` + hard-refresh to pick up the embedded web asset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…don't jitter
- The per-cell bake stage now names the chart in flight ("Baking US5MD1MD" instead of a
generic "Baking charts"): ComposeENCRoot's progress callback passes the cell stem, and
composeProvider puts it in the action. The CLI prints it too.
- The server reports the ETA as its OWN field, separate from the count `detail`, so the
client renders each in a fixed, right-aligned tabular slot (the ETA reserves a min-width).
Glued together they changed width every tick and the whole line shifted around; split, the
count and ETA stay put. The chart-library batch banner re-joins them for its one-line layout.
Rebuild with `make TILE57=../tile57 build` + hard-refresh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Composer TileSource backed by tile57's on-demand compositor: per-cell
PMTiles held mmap'd + the ownership partition resident, composing each tile as
the camera asks instead of serving a prebaked archive. It returns decompressed
MLT (the HTTP layer gzips on the wire) with TileType=mlt.
A dev/test hook wires it in: set TILE57_LIVE_COMPOSE to a dir of per-cell
archives (from `tile57 compose --keep-cells`), optionally TILE57_LIVE_PARTITION
to a sidecar (`--save-partition`) and TILE57_LIVE_NAME (default "live"), and the
set is served at /tiles/{name}/{z}/{x}/{y}. No-op when unset.
Verified against NOAA d05 (848 cells): the server opens the compositor at boot
(loads the partition sidecar), and GET /tiles/live/14/4706/6244 returns the
3480-byte MLT byte-matching the batch reference — 0.61 ms/tile end-to-end over
HTTP, with gzip-on-the-wire and 204 for out-of-coverage tiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…as map layers
handlePacks built the provider list from disk packs + ENC_ROOT dirs only, so a
set that lives only in the s.sets registry (the live runtime compositor) never
reached the client and never became a map layer. Include s.sets.names() in the
union. Everything downstream already works for a registry set: the TileJSON
(/tiles/{name}.json) derives bounds/zoom/encoding from src.Meta(), and the engine
style defaults to all registered sets — so the client creates the chart-<name>
source and the style supplies its layers.
Verified: with TILE57_LIVE_COMPOSE set, /api/packs lists "live", and the web map
renders the Chesapeake from on-demand composed tiles (full S-52 symbology).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ict compose) Importing a provider now builds the live compose structure itself — no CLI, no env vars. bakeProvider bakes each cell to a KEPT per-provider dir (<setDir>/cells-pm, incremental: adding a district re-bakes only its new cells), opens a runtime Composer over them, and saves the ownership-partition sidecar (<setDir>/partition. tpart) — skipping the whole-district compose pass (tiles compose on demand). The provider registers + renders exactly like a baked pack. On boot, registerLive Providers re-opens a Composer for every provider with kept archives (loading the sidecar → ~25ms), so live layers survive a restart; rebakeMissingProviders skips providers already served. TILE57_BATCH_COMPOSE=1 keeps the portable chart.pmtiles path; a tile57-engine-commit stamp invalidates stale per-cell archives on upgrade. Refactors: registerBakedSet → registerProviderSet(src, packPath) so a live source (no disk pmtiles) registers with the same tail (cell manifest, aux, meta); baker. PrepareLive is the input-prep half; tilesource.Composer.SavePartition persists the sidecar. Fix: scanPacks now skips the cells-pm dir — its per-cell PMTiles are compositor INPUTS, not packs, so they no longer each surface as a provider (which made the client probe /tiles/<cell>.json → 404 spam). Verified on NOAA d05 (848 cells): boot registers one "noaa" live provider (z0..16), /api/packs lists it with district d05, and the web map renders the Chesapeake from on-demand composed tiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use the ownership partition to decide caching, so a live provider never caches a
blank that will fill in. The compositor reports `owned` per tile (a cell's face
covers it → SHOULD render). tileserve then:
- content, or unowned blank (true empty ocean) → immutable per ?g;
- owned-but-blank (a cell owns it but produced nothing), or the set is still
baking (imports.runningFor) → no-cache, so it re-fetches once the cell lands;
- owned-but-blank AFTER bakes are done → logged as a likely missing/failed cell.
Generation token: a live provider now carries a ?g (live.gen, bumped on each
completed import via bumpLiveGen; packGen falls back to it), so the client
re-fetches with a fresh URL when the cell set changes — invalidating tiles it
cached against a previous, less-complete set. registerLiveProviders only re-serves
a provider whose import completed (live.gen present); a set left partial by an
interrupted bake is finished by rebakeMissingProviders instead of served partial.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n path (host) Step A of the legacy cleanup — remove every host caller of the batch/bundle tile-production paths so cell-bake + live-compose is the only route: - bakeProvider collapses to prepareLiveProvider → registerProviderSet; drop the TILE57_LEGACY_BAKE (BakeBundle) and TILE57_BATCH_COMPOSE (composeProvider) branches. - delete composeProvider, registerBakedSet (the chart.pmtiles opener), bakeProgress (BakeBundle's per-band bar), and baker.ComposeENCRoot (BakeCell → ComposeFiles batch). - CLI `bake` now writes the LIVE STRUCTURE: <out>/tiles/*.pmtiles (per cell, via baker.PrepareLive, incremental) + <out>/partition.tpart (NewComposer + SavePartition). No merged chart.pmtiles, no per-scheme style/assets/manifest emission — style + assets are served at runtime. Deletes runTile57Archive/runTile57Bundle/bakeTile57Bundle + writeManifestJSON. Nothing here calls tile57.BakeBundle/BakePmtiles/Compose/ComposeFiles anymore — the Go bindings + C ABI for those come out next (Steps B–D). Builds + vets clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Align the runtime compositor's kept per-cell archives with the layout the `tile57 bake -o <dir>` CLI writes (<dir>/tiles/<STEM>.pmtiles + partition.tpart), so a CLI-baked structure drops straight into a provider's set dir. liveCellsDir now returns <setDir>/tiles. scanPacks recognizes a live provider by its partition.tpart sidecar and skips the whole tiles/ dir (its per-cell archives are compositor INPUTS, not packs) — else every cell would surface as a phantom provider. A legacy batch bundle's tiles/chart.pmtiles (no sidecar) still registers by provider name. Tests: new TestScanPacksSkipsLiveProvider locks the skip/keep behavior; the stale TestImportPacks (asserted the removed batch tiles/chart.pmtiles) now asserts the live structure — per-cell tiles/<STEM>.pmtiles + partition.tpart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndlers A live runtime-compositor provider has no disk pack (it lives only in the set registry), so several handlers that assumed a packPath left it half-supported. Surface it provider-centrically instead: - /api/packs: emit coverage bounds for a live provider from the registry when enabled, else from its meta sidecar (BBox) — not only for disk packs. - /api/set/enable: re-register a disabled live provider by re-opening its runtime compositor from the kept per-cell archives, not just re-opening a disk pack (previously disable->enable dropped a live set until restart). - enabledPackCells (?active): include live providers' cells from the same per-provider cell manifest, so their charts count as "on the map". TestImportPacks now also asserts the live provider's bounds appear in /api/packs and that disable->enable round-trips through the registry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_setLabel branded district / river sets (noaa-d5 -> "NOAA · Mid-Atlantic",
ienc-* -> "IENC · …") but a BARE provider set — which the live-composite model
registers, one set per provider ("noaa") — fell through to the raw lowercase
routing key. Map it to the branded display name from the existing _providers()
list ("noaa" -> "NOAA", "ienc" -> "Inland ENC", "user" -> "User Charts"). The
routing key stays lowercase; this is display only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… dir Make provider discovery fully provider-centric. scanPacks now reads ONLY the flat <cache>/tiles dir for hand-dropped standalone .pmtiles/.mbtiles archives, instead of walking the whole cache. Live runtime-compositor providers own <cache>/<PROVIDER>/tiles/*.pmtiles + partition.tpart and are discovered by registerLiveProviders (iterating installed providers) — never scavenged here. This supersedes the earlier partition.tpart-guarded skip with a stronger invariant: an interrupted import (per-cell archives present, partition not saved yet) can no longer leak its cells as phantom sets, because provider dirs aren't scanned at all. TestScanPacksStandaloneArchivesOnly covers flat discovery + the provider-input skip, both with and without a saved partition sidecar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the live provider's live.gen MTIME cache-bust token with a CONTENT token: a sha-of-shas over its per-cell archives. PrepareLive writes an <archive>.sha sidecar (the archive's content sha) at bake; liveGenToken sorts the per-cell "stem:sha" lines and hashes them into a positive int63, stored in live.gen and read by packGen. So a no-op re-bake keeps the token (the client's cached tiles stay valid), and the token advances exactly when the cell set or a cell's content changes — the foundation for progressive per-batch re-keying during import. TestLiveGenTokenContentAddressed covers determinism, content-sensitivity, and add/remove. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…p live Register + serve a live provider DURING its import instead of only when the whole provider finishes: every liveReKeyBatch (48) freshly-baked cells, re-open the compositor over what's baked so far, register + enable it, and advance the content token (progressiveReKey, driven from the bake progress callback). The client re-fetches on each new ?g and the map fills in batch by batch. The per-re-key partition rebuild is negligible against the per-cell (seconds-each) bake, and it runs synchronously in the paused bake loop so it never races the writer. openLiveComposer now ALWAYS persists the (possibly rebuilt) partition sidecar, so a re-key over a changed cell set leaves the on-disk sidecar current for a fast boot instead of stale. TestImportPacks exercises progressiveReKey over the real baked archives. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a tile57 Rewire the live-provider bake onto tile57.BakeTree: one call walks the ENC tree and bakes every cell IN PARALLEL to the mirrored <cellsDir>/<rel>.pmtiles (+ .sha), the engine writing + freeing each archive (peak memory ~ workers). This restores parallelism after making the cell the sole parallel unit (the host had gone fully serial). - PrepareLive is now a thin BakeTree wrapper (drops the serial per-cell loop, the .sha write, and ListCells-based iteration — the library does all of it). - liveCellArchives walks the mirrored tree (was a flat ReadDir). - liveBakeWorkers bounds concurrency (min(cpus,8), CHARTPLOTTER_BAKE_WORKERS override). - Progress flows through BakeTree's (done,total) callback; drop the mid-bake progressive re-key (each re-key rebuilt the partition → StalePartition churn) — the provider registers once at the end. - bakeProvider's failed-import cleanup removes only the baked tiles dir, not the whole set dir (which holds ENC_ROOT in single-dir mode) — fixes TestImportFetchDownloadOnly. - TestImportPacks asserts the cell's archive in the mirrored tree. Server package green. A cell shared by two districts now bakes once per district dir (the ownership partition tie-breaks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Live PrepareLive's signature changed (workers + (done,total) progress, returns a count). The CLI now bakes in parallel and walks <out>/tiles for the mirrored per-cell archives to build the partition sidecar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sBytes, ComposeSource.Tile Cell-index and bake-metadata scans call tile57.Cells(path) directly (no chart handle over raw S-57 anymore); the simulator water mask uses tile57.FeaturesBytes; the composer serves via ComposeSource.Tile. Submodule at tile57 599f8ad (bake/render/compose sections, PMTiles-only mmap'd chart handle, compose borrows charts, tile57_status everywhere). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…MBOL_SCALE icons), sector figures whole across seams, compose output backends, tile57_free(ptr) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bake required for decoration) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rt metadata' phase no longer holds the catalogue in RAM providerCellData loaded every base cell + update of the whole provider into memory (~8 GB for the full NOAA catalogue), only for ExtractCellMeta to write it all back to a temp dir and re-scan it. registerProviderSet now reads the per-cell metadata straight from the provider's on-disk ENC_ROOT (baker.ExtractCellMetaDir over tile57.Cells — the engine streams cells and never holds the set), and the cell manifest is written from the resulting stems. The zip-upload path keeps its in-memory CellData (district-sized, already in RAM from the upload). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…acks (kills false IALA boundaries at cell junctions) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… coarse zooms, scamin-restricted; re-bake required) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… build registerLiveProviders re-registered kept per-cell archives at boot without the .enginever check (that only ran on the import path), so a binary upgrade kept serving stale tiles until a manual re-import. Skip stale sets at boot — rebakeMissingProviders then re-bakes them through prepareLiveProvider, which drops + re-stamps the dir. Also bumps the tile57 engine to 3c46b23 (sector- light fixes: sub-band SCAMIN cull, 512-px figure sizing, compose reach ring). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live compositor generates tiles at SERVE time, so the ?g content token must address the engine build too: a serve-path engine fix used to leave the token unchanged (archives byte-identical), and clients kept their cached broken tiles until a manual hard refresh. liveGenToken now mixes Server.EngineCommit into the sha-of-shas, and registerLiveProviders re-persists live.gen at boot so a restart under a new engine mints the new token without waiting for the next import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…es, submodule to b8e8499 The engine renamed its public surface so every symbol leads with its family (tile57_chart_* handle methods, bake_charts/enc_charts, Go Cells -> Charts, BakeCell -> BakeChart). Host call sites follow; host-internal Cells vocabulary (import packs, set meta, ienc) is untouched. Binary rebuilt against the new libtile57.a (engineCommit b8e8499, tile57_chart_* symbols verified in the binary); baker/tilesource/server tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s it as an arg
registerLiveProviders and rebakeMissingProviders run inside server.New, but
EngineCommit was assigned by the caller AFTER New returned — so the .enginever
stamp check always ran against "" ("counts as current") and stale kept
archives registered on every boot, silencing the self-heal re-bake. This is
why an engine upgrade never re-baked the live NOAA set: tiles baked pre-
meta-bounds-complement (and interrupted at d1+d5 of five districts) kept
serving. The commit is construction-time data, so it is now a New parameter,
set before boot registration; serve.go passes the ldflags stamp, tests pass "".
Also bumps the engine submodule to aeac905 (tile57_version now reports 0.3.0 —
the capi string had missed the version bump).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tiles serve throughout An engine upgrade used to RemoveAll the kept per-cell archives up front and re-bake in place: an interrupted or failed re-bake left the provider with no tiles at all (and boot refused to serve the stale survivors, blanking the map for the whole multi-minute bake). Now stale archives keep serving — stale tiles beat no tiles — while the full re-bake lands in a staging tree (<setDir>/tiles.next, stamped with the baking engine up front so a same-build re-run resumes incrementally and a different build starts over). The swap into place is the LAST step, after the whole bake succeeded; a staged bake that produces nothing is discarded. registerProviderSet's re-register closes the old composer, so the dropped archives' mmaps release on swap. Boot recovers the one crash window (old tree dropped, staged tree not yet renamed) by moving the staged tree into place and routing the set through the self-heal bake. invalidateLiveOnEngineChange is gone; rebakeMissingProviders now also queues sets that are serving from another engine build's archives. Covered by TestEngineRebakeStagesAndSwapsLast (failure path keeps the served tree untouched; success path swaps, restamps, and leaves no staging behind). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ule) Four files weren't gofmt-clean (Go 1.26 toolchain gofmt). Also scope `make fmt` /`fmt-check` to `git ls-files '*.go'` instead of `.` — the latter walked into the ./tile57 engine submodule and failed the gate on the engine's own Go bindings, which chartplotter-go doesn't own. git ls-files lists only this repo's tracked files, so the submodule is naturally excluded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
govulncheck flagged GO-2026-5856 (Encrypted Client Hello privacy leak in crypto/tls), reached from serveOSM's HTTP client. Fixed in the standard library as of go1.26.5. Bump the toolchain directive; govulncheck is clean afterward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…GO-2026-5856) go.mod already pins `toolchain go1.26.5`, but CI ignored it: setup-go `go-version: '1.26'` installed the cached 1.26.4 and GOTOOLCHAIN=local blocks the auto-upgrade to the toolchain directive, so govulncheck ran under 1.26.4 and flagged GO-2026-5856. Pin the exact patch (1.26.5) in every workflow; release.yml especially, so shipped binaries link the patched crypto/tls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d73c) The old pin (aeac905c8) predates the cross-platform filemap.zig refactor, so a non-floating `submodule update --init` checkout still hit the `std.posix.mmap` void-flag break on the windows cross-compile. Bump the last-known-good pin to 4b6d73c (feat/per-cell-composite tip, now merged to tile57 main), which builds clean for x86_64-windows-gnu. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.