Store path settings (hash_prefix/schema_prefix) were not honored by writers or GC — fixed in 2.3.1

[dimitri-yatsenko]

Summary

The per-store path settings hash_prefix and schema_prefix (documented in Configure Object Storage as controlling where each storage section lives) were not incorporated by the storage writers or garbage collection in DataJoint 2.x through 2.3.0:

• Hash-addressed writes (<blob@>, <attach@>, <hash@>) hardcoded the _hash/{schema}/… layout — a configured hash_prefix was accepted and validated but silently ignored.
• Schema-addressed writes (<object@>, <npy@>, plugin SchemaCodecs) ignored schema_prefix entirely and wrote to root-level {schema}/{table}/… paths instead of the documented _schema/{schema}/{table}/… section.
• Garbage collection likewise hardcoded _hash/ and enumerated root-level paths.
• The only consumer of the settings was <filepath@> insert-time validation, which reserved the declared values — so overriding hash_prefix even un-reserved the real _hash/ namespace.

Fixed in 2.3.1 (PR #1479)

The settings are now honored end to end from a single source (get_store_spec, defaults _hash / _schema applied to every spec including plugin protocols; no component-level fallbacks): writers place objects under the configured sections, GC scans the configured hash section and never enters the configured filepath_prefix namespace, and <filepath@> validation reserves the same values the writer uses.

Impact — upgrading may require checking paths

1. Default configuration (most pipelines). Hash-addressed layout is unchanged (_hash/…). Schema-addressed objects: new writes land under _schema/{schema}/…, while objects written by ≤2.3.0 remain at root-level {schema}/…. Both remain fully readable (every row's metadata records its complete path) and GC lists both layouts, so mixed stores are handled. But anything outside DataJoint that assumed the root-level layout — sync jobs, backup filters, bucket policies, manual browsing — should be checked and, where possible, switched to reading paths from row metadata (ObjectRef.path) rather than constructing them.
2. Stores configured with custom hash_prefix/schema_prefix before 2.3.1. Those settings did nothing at write time, so existing objects live at the defaults while the config declares otherwise. After upgrading, new writes will move to the configured prefixes, splitting the store's layout, and GC scans only the currently configured hash section — objects under the old section are not reclamation candidates while the custom setting is active, and hash deduplication will not match content across sections. Recommended: verify what your store actually contains before upgrading, and either drop the custom prefixes (keep the defaults that match your existing data) or accept the split knowingly.
3. Do not change prefixes on a store that already holds data — now documented with a warning (datajoint/datajoint-docs#196).

Related

• Store keys hash_prefix/schema_prefix are accepted but not honored by writers or GC #1486 — the design-level record of the validate-and-ignore state (resolved the same way; closes with PR fix(#1469): codec-driven GC reference discovery + file-level orphan matching #1479).
• docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics datajoint-docs#196 — documentation updated to the settings-driven layout, per-schema hash paths, and GC semantics.