Skip to content

Fix NixIO writing of datetime array_annotations (#1198)#1875

Open
Leonard013 wants to merge 1 commit into
NeuralEnsemble:masterfrom
Leonard013:fix/1198-nixio-datetime-array-annotations
Open

Fix NixIO writing of datetime array_annotations (#1198)#1875
Leonard013 wants to merge 1 commit into
NeuralEnsemble:masterfrom
Leonard013:fix/1198-nixio-datetime-array-annotations

Conversation

@Leonard013

Copy link
Copy Markdown

Fixes #1198.

Why

NixIO cannot write array_annotations whose elements are datetime (or
date/time) objects. neo.core accepts such values as array-annotation
elements (they are in ALLOWED_ANNOTATION_TYPES, stored as an object-dtype
ndarray), so a Segment/Event/AnalogSignal carrying them can be built and
lives fine in memory — but writing it to a .nix file crashes:

File ".../neo/io/nixio.py", line 1283, in _write_property
    section.create_property(name, values)
File ".../nixio/section.py", line 156, in create_property
    dtype = DataType.get_dtype(single_val)
File ".../nixio/datatype.py", line 50, in get_dtype
    raise ValueError("Unknown type for value {}".format(value))
ValueError: Unknown type for value 2021-01-01 00:00:00

As @JuliaSprenger pointed out in the issue, _write_property must convert the
datetime objects before saving. Neo already knows how to do this for scalar
datetime annotations via dt_to_nix() / dt_from_nix(); the iterable
(array-annotation) branch just never used them.

What

neo/io/nixio.py, two surgical changes:

  1. Write (_write_property, iterable branch): when an item is a
    datetime/date/time, convert it to its ISO string with dt_to_nix()
    and record the returned annotation type as the property's definition
    exactly mirroring the existing scalar-annotation path a few lines above.
  2. Read (_nix_attr_to_neo): when a property carries a datetime
    definition and holds multiple values (a list), reconstruct it
    element-wise with dt_from_nix() into an object-dtype ndarray. A single
    value keeps the previous scalar behaviour unchanged. The object dtype is
    required so the reconstructed annotation is accepted back by
    array_annotate() on the read path.

This makes datetime array_annotations survive a write → read roundtrip for all
three of datetime, date and time.

Tests

Added NixIOWriteTest.test_datetime_array_annotations in
neo/test/iotest/test_nixio.py (modelled on the neighbouring
test_empty_array_annotations): writes an Event with datetime, date and
time array_annotations, reads it back, and asserts equality. The test
fails on master (same ValueError: Unknown type for value ...) and
passes with this change.

Verification (Python 3.12, nixio 1.5.4, numpy 2.5.1):

black --check (changed files)   -> 2 files left unchanged
pytest new regression test      -> 1 passed
pytest full test_nixio.py       -> 36 passed, 7 skipped  (skips need datalad web downloads)
pytest core dataobject          -> 5 passed
source-only revert of the fix   -> the new test fails with the ValueError above

Scope / caveats for reviewers

  • numpy datetime64 (the literal issue title) is intentionally out of
    scope.
    neo core rejects numpy.datetime64 before it can reach any IO
    (ALLOWED_ANNOTATION_TYPES in neo/core/baseneo.py does not include
    np.datetime64), so the values that actually reach _write_property are
    python datetime objects held in an object-dtype array — which is what the
    issue's own diagnosis ("convert datetime objects before saving") refers to.
    Broadening core validation to accept datetime64 would be a separate, larger
    change.
  • The read path distinguishes "array" vs "scalar" datetime purely by
    isinstance(values, list). A length-1 datetime array_annotation is stored and
    read back as a scalar, then normalised to a 1-element object array by
    array_annotate — verified to roundtrip correctly.
  • Not touched: the nixrawio/lazy read path — array_annotations roundtrip is a
    full-NixIO feature.

Prepared with AI assistance (Claude Code): the reproduction, fix, and
regression test were AI-drafted, and the reasoning and verification are
documented in full above. Happy to adjust the approach or scope.

NixIO._write_property passed datetime/date/time objects contained in
array_annotations straight to nixio.create_property, which cannot store
them and raised "Unknown type for value ...". Convert each datetime item
to its ISO string form via dt_to_nix() on write (mirroring the existing
scalar-annotation path) and record the annotation type as the property
definition. On read, reconstruct multi-element datetime array_annotations
element-wise into an object-dtype ndarray so the write->read roundtrip
preserves them.

Fixes NeuralEnsemble#1198

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NixIO does not support writing of numpy array_annotations of dtype datetime

1 participant